File Desciptor Limit Preventing Connections
Problem
In Linux it is not uncommon for high traffic applications to consume all available file descriptors and present like a connectivity issue
Should you see error about connectivity that have no apparent cause, look for repeating errors like this in your catalina.out
24-Apr-2021 20:00:57.691 SEVERE [http-nio-28080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:455)
at java.lang.Thread.run(Thread.java:748)
Solution
Get the PID of your process (Line 2: “1664” in the block below)
[codeglitch@localhost ~]$ ps -ef | grep activemq
activemq 1664 1 0 May03 ? 00:01:44 /usr/bin/java -Xms64M -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/activemq//conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/activemq//tmp -Dactivemq.classpath=/opt/activemq//conf:/opt/activemq//../lib/: -Dactivemq.home=/opt/activemq/ -Dactivemq.base=/opt/activemq/ -Dactivemq.conf=/opt/activemq//conf -Dactivemq.data=/opt/activemq//data -jar /opt/activemq//bin/activemq.jar start
codegli+ 33170 29722 0 21:29 pts/0 00:00:00 grep --color=auto activemq
[codeglitch@localhost ~]$
Check the file descriptor limit for the process in question (line 10: “Max open files” in the block below)
[codeglitch@localhost ~]$ cat /proc/1664/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 11060 11060 processes
Max open files 262144 262144 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 11060 11060 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
[codeglitch@localhost ~]$
Use lsof
to determine the number of file descriptors your process is using. (line 4: “314” in the block below)
[codeglitch@localhost ~]$ sudo lsof -a -p 1664 | wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
314
[codeglitch@localhost ~]$
Adjust the file descriptor limit according to the instructions for your Linux Disto / Version