IBM Too Many Open Files error message - United States wasrun; wasjdk; open; files; too; ProblemDeterminationDocument; JCC was appserver app server TECHNOTE (TROUBLESHOOTING)

PROBLEM(ABSTRACT)
 This technote explains how to debug the "Too many open files" error message on Microsoft Windows, AIX, Linux and Solaris operating systems. 

SYMPTOM
The following messages could be displayed when the process has exhausted the file handle limit: 

java.io.IOException: Too many open files

[3/14/15 9:26:53:589 EDT] 14142136 prefs W Could not lock User prefs. Unix error code 24.

New sockets/file descriptors can not be opened after the limit has been reached. 


CAUSE
System configuration limitation.

When the "Too Many Open Files" error message is written to the logs, it indicates that all available file handles for the process have been used (this includes sockets as well). In a majority of cases, this is the result of file handles being leaked by some part of the application. This technote explains how to collect output that identifies what file handles are in use at the time of the error condition.


RESOLVING THE PROBLEM




Determine Ulimits

On UNIX and Linux operating systems, the ulimit for the number of file handles can be configured, and it is usually set too low by default. Increasing this ulimit to 8000 is usually sufficient for normal runtime, but this depends on your applications and your file/socket usage. Additionally, file descriptor leaks can still occur even with a high value. 

Display the current soft limit:
ulimit -Sn

Display the current hard limit:
ulimit -Hn

Or capture a Javacore, the limit will be listed in that file under the name NOFILE:
kill -3 PID


Please see the following document if you would like more information on where you can edit ulimits:
Guidelines for setting ulimits (WebSphere Application Server)
http://www.IBM.com/support/docview.wss?rs=180&uid=swg21469413 [http://www.ibm.com/support/docview.wss?rs=180&uid=swg21469413]


Operating Systems 

Windows By default, Windows does not ship with a tool to debug this type of problem. Instead Microsoft provides a tool that you can download called Process Explorer [http://technet.microsoft.com/en-us/sysinternals/bb896653]. This tool identifies the open handles/files associated with the Java™ process (but usually not sockets opened by the Winsock component) and determines which handles are still opened. These handles result in the "Too many open files" error message. 

To display the handles, click on the Gear Icon [/support/docview.wss?uid=swg21067352&amp;aid=1] in the toolbar (or press CTRL+ H to toggle the handles view). The icon will change to the icon for DLL files (so you can toggle it back to the DLL view). 

It is important that you change the Refresh Rate. Select View > Update Speed, and change it to 5 seconds. 

[/support/docview.wss?uid=swg21067352&aid=2] [/support/docview.wss?uid=swg21067352&aid=2] 

There is also another Microsoft utility called Handle that you can download from the following URL: 
https://technet.microsoft.com/en-us/sysinternals/bb896655.aspx [https://technet.microsoft.com/en-us/sysinternals/bb896655.aspx] 

This tool is a command line version of Process Explorer. The URL above contains the usage instructions. 


AIX 
The commands lsof and procfiles are usually the best commands to determine what files and sockets are opened.. 

lsof 
To determine if the number of open files is growing over a period of time, issue lsof to report the open files against a PID on a periodic basis. For example: 

lsof -p [PID] -r [interval in seconds, 1800 for 30 minutes] > lsof.out 

This output does not give the actual file names to which the handles are open. It provides only the name of the file system (directory) in which they are contained. The lsof command indicates if the open file is associated with an open socket or a file. When it references a file, it identifies the file system and the inode, not the file name. 

It is best to capture lsof several times to see the rate of growth in the file descriptors. 

procfiles 
The procfiles command does provide similar information, and also displays the full filenames loaded. It may not show sockets in use. 

procfiles -n [PID] > procfiles.out 


Other commands (to display filenames that are opened) 

INODES and DF 
df -kP filesystem_from_lsof | awk '{print $6}' | tail -1 
>> Note the filesystem name 
find filesystem_name -inum inode_from_lsof -print > filelist.out 
>> Shows the actual file name 

svmon 
svmon -P PID -m | grep pers (for JFS) 
svmon -P PID -m | grep clnt (for JFS2, NFS) 
(this opens files in the format: filesystem_device:inode) 

Use the same procedure as above for finding the actual file name. 


Linux To determine if the number of open files is growing over a period of time, issue lsof to report the open files against a PID on a periodic basis. For example: 

lsof -p [PID] -r [interval in seconds, 1800 for 30 minutes] > lsof.out 

The output will provide you with all of the open files for the specified PID. You will be able to determine which files are opened and which files are growing over time. 

It is best to capture lsof several times to see the rate of growth in the file descriptors. 


Alternately you can list the contents of the file descriptors as a list of symbolic links in the following directory, where you replace PID with the process ID. This is especially useful if you don't have access to the lsof command: 

ls -al /proc/PID/fd 


Solaris Run the following commands to monitor open file (socket) descriptors on Solaris: 

pfiles 
/usr/proc/bin/pfiles [ PID ] > pfiles.out 

lsof 
lsof -p [ PID ] > lsof.ou t 

This will get one round of lsof output. If you want to determine if the number of open files is growing over time, you can issue the command with the -r option to capture multiple intervals: 
lsof -p [PID] -r [interval in seconds, 1800 for 30 minutes] > lsof.out 

It is best to capture lsof several times to see the rate of growth in the file descriptors. 

HP-UX 
lsof 
lsof -p [ PID ] > lsof.out 

This will get one round of lsof output. If you want to determine if the number of open files is growing over time, you can issue the command with the -r option to capture multiple intervals: 
lsof -p [PID] -r [interval in seconds, 1800 for 30 minutes] > lsof.out 

It is best to capture lsof several times to see the rate of growth in the file descriptors. 



Cross reference information Segment Product Component Platform Version Edition Application Servers Runtimes for Java Technology Java SDK
Too many open files error cause Portal server out of service after a restart due to Hardware failure. After experienced a Hardware problem (OS shutdown totally), client start Portal Server and run for some time. Then they found the Portal out of service. The backend log shows lots of error related with Too Many Open Files
On UNIX and Linux operating systems, the ulimit for the number of file handles can be configured, and it is usually set too low by default. Increasing this ulimit to 8000 is usually sufficient for normal runtime, but this depends on your applications and your file/socket usage. Additionally, file descriptor leaks can still occur even with a high value. 

Display the current soft limit:
ulimit -Sn

Display the current hard limit:
ulimit -Hn

Or capture a Javacore, the limit will be listed in that file under the name NOFILE:
kill -3 PID