Indexer Paused with "Too many open files" Error
While indexing a repository the indexer stops and the following error can be seen in the logs (example below for an SVN repository):
ERROR [FishEye SVN Diff Fetcher] fisheye.app DiffProcessor-getRequiredInfo - SVN Error diffing revision 4127 com.cenqua.fisheye.rep.RepositoryClientException: org.tigris.subversion.javahl.ClientException: svn: Can't read length line from file /artimi/db/revs/138: /artimi/db/revs/138 (Too many open files) at com.cenqua.fisheye.svn.SvnThrottledClient.diff(SvnThrottledClient.java:119) at com.cenqua.fisheye.svn.diff.DiffProcessor.doDiff(DiffProcessor.java:55) at com.cenqua.fisheye.svn.diff.DiffProcessor.processChangePaths(DiffProcessor.java:194) at com.cenqua.fisheye.svn.diff.DiffProcessor.getRequiredInfo(DiffProcessor.java:317) at com.cenqua.fisheye.svn.diff.DiffProcessor.processMessage(DiffProcessor.java:268) at com.cenqua.fisheye.svn.diff.DiffFetcher.process(DiffFetcher.java:64) at com.cenqua.fisheye.svn.diff.DiffFetcher.run(DiffFetcher.java:45) at java.lang.Thread.run(Unknown Source) Caused by: org.tigris.subversion.javahl.ClientException: svn: Can't read length line from file /artimi/db/revs/138: /artimi/db/revs/138 (Too many open files) at org.tigris.subversion.javahl.JavaHLObjectFactory.throwException(JavaHLObjectFactory.java:435) at org.tmatesoft.svn.core.javahl.SVNClientImpl.throwException(SVNClientImpl.java:1309) at org.tmatesoft.svn.core.javahl.SVNClientImpl.diff(SVNClientImpl.java:783) at org.tigris.subversion.javahl.SVNClient.diff(SVNClient.java:206) at com.cenqua.fisheye.svn.SvnThrottledClient.diff(SvnThrottledClient.java:116) ... 7 more
The indexer has hit the the maximum permitted number of open files in Linux/UNIX operating systems.
The current values for open file descriptors and maximum allowed file descriptors can be determined by turning on debug logging under
Administration >> Global Settings >> Server >> Debug Logging
After debug logging as been turned on, the
Resource Monitor will provide values for
ofd (open file descriptors) and
mfd (maximum file descriptors):
DEBUG [Resource Monitor ] fisheye SystemResourceMonitor$MonitoringTask-run - tc=243, ofd=4080, mfd=4096, fm=114635672, tm=847773696
The above output indicates that Fisheye can have a maximum of 4096 file descriptors and currently is using 4080 file descriptors.
There are two possible resolutions:
- Optimize the Search Indexes on restart.
More information can be found on the following page: Optimizing the Search Indexes on Restart
Increase the number of allowed file handles per process (and possibly for the entire system).
There are two file handle limits that can affect Fisheye/Crucible: the total number of file handles allowed per process and the number of file handles allowed on the entire system.
Fisheye/Crucible can hold a high number of open file handles which often exceeds common per-process default limits – usually between 1000 and 4000 open files. If these limits are reached we recommend increasing the limit to 40000. The total number of file handles allowed on the entire system is usually set high enough by default but on an server with many other running applications this too may need to be increased.
These settings may vary between operating systems so please check with a system administrator or perform additional research before making any changes. Described below is the most common approach to troubleshooting and fixing this problem.
Setting the permitted number of file handles per process
To check the number of currently open file handles, run
sudo lsof -u FISHEYE_USER(restricted to the FisheEye running user) or
sudo lsof -p FISHEYE_PROCESS(restricted to the fisheye process).
To check the permitted number of file handles per process, run
To modify the permitted number of file handles per process, edit
/etc/security/limits.confand add (replace FISHEYE_USER with the name of the user that will run Fisheye/Crucible or use * to make the change for all users):
Single user: FISHEYE_USERIt is recommended to set the limits for the
FISHEYE_USERexplicitly first before increasing the limit for all users.
FISHEYE_USER soft nofile 40000 FISHEYE_USER hard nofile 40000
* soft nofile 40000 * hard nofile 40000
Setting the permitted number of file handles over the whole system
To check the number of open file handles, run
To check the total number of open file permitted in the whole system, run
To modify the total number of open file permitted in the whole system, edit
/etc/sysctl.confand add the line:
fs.file-max = 40000
Note: After making these changes the OS will need to be restarted. For further information please see this page.
The Ubuntu server requires additional additional steps in order have the larger limits register (based on this article): add “session required pam_limits.so” in
sudo vi /etc/pam.d/common-session
session required pam_limits.soto the end of the file.
- Reboot the OS.
If Fisheye/Crucible is running under j2ee, add the ulimit into Fisheye/Crucible's daemontool runscript.
ulimit -n 40000