Problem

In Linux it is not uncommon for high traffic applications to consume all available file descriptors and present like a connectivity issue

Should you see error about connectivity that have no apparent cause, look for repeating errors like this in your catalina.out

24-Apr-2021 20:00:57.691 SEVERE [http-nio-28080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
 java.io.IOException: Too many open files
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:455)
	at java.lang.Thread.run(Thread.java:748)
CODE

Solution

Get the PID of your process (Line 2: “1664” in the block below)

[codeglitch@localhost ~]$ ps -ef | grep activemq
activemq    1664       1  0 May03 ?        00:01:44 /usr/bin/java -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/activemq//conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/activemq//tmp -Dactivemq.classpath=/opt/activemq//conf:/opt/activemq//../lib/: -Dactivemq.home=/opt/activemq/ -Dactivemq.base=/opt/activemq/ -Dactivemq.conf=/opt/activemq//conf -Dactivemq.data=/opt/activemq//data -jar /opt/activemq//bin/activemq.jar start
codegli+   33170   29722  0 21:29 pts/0    00:00:00 grep --color=auto activemq
[codeglitch@localhost ~]$
CODE

Check the file descriptor limit for the process in question (line 10: “Max open files” in the block below)

[codeglitch@localhost ~]$ cat /proc/1664/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             11060                11060                processes 
Max open files            262144               262144               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       11060                11060                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
[codeglitch@localhost ~]$
CODE

Use lsof to determine the number of file descriptors your process is using. (line 4: “314” in the block below)

[codeglitch@localhost ~]$ sudo lsof -a -p 1664 | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
314
[codeglitch@localhost ~]$
CODE

Adjust the file descriptor limit according to the instructions for your Linux Disto / Version