Too many open files in system errors on Bamboo or Bamboo Remote Agents hosted on Linux
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
Error “Too many open files in system” is written to the logs when running Bamboo or a Bamboo Remote Agent.
2021-04-26 10:00:00,000 ERROR \[0-BAM::ln-dockerbuild01.dev.atlassian.com: 1::Agent:pool-3-thread-1] \[BuildLogFileWriter]
java.io.FileNotFoundException: <path> \(Too many open files in system\)
at java.io.FileOutputStream.open0(Native Method\)
Ulimit setting or a single user issue
If the command below will show the value for "open files" and "max user processes." If this value is set to 16384 or lower the system could log the error and the build will fail. The default for this setting in most linux distributions is between 1024 and 4096.
sudo -u <atlbitbucket> bash -c "ulimit -n -u"
System value or if ulimit is set high
For the system, Please see your Linux distro for the default value. This error can be logged when multiple users have many open files. For instance, if using docker all of the files are opened as different users. So the number of files can increase significantly. This is how to find the current settings.
For the user environment nofile is set too low, use solution Ulimit Settings. Or for the system fs.file-max is set too low, use solution File-max.
Note that the max value for file-max, at time of writing, is 2147483584 on x86_64. If not using x86_64, please see your documentation for the max value.
Remote Agent had a "Too many open files in system errors" in the system logs. You are looking for the recommend ulimits settings for remote agents.
Our current recommendation to disable this check by setting the value very high for the user running the Remote Agent for noproc and nofile. We suggest to modify the permitted number of file handles per process, edit /etc/security/limits.conf and add or edit the following lines:
Bamboo_USER soft noproc 16384
Bamboo_USER hard noproc 16384
Bamboo_USER soft nofile 16384
Bamboo_USER hard nofile 16384
Replacing Bamboo_USER with the name of the user running Bamboo. Please note that noproc is the number of processes that a user may open. This is a similar setting to nofile but can be hard to diagnose if hitting this value. A user will need to be logged out and logged back in for all login environments for this to take effect, stopping and starting may not be enough. If unsure, please reboot the system.
For the system value needs to be high enough for all users of the system. On a current Ubuntu system the default is 6549203. This is usually high enough. But if you run into this issue, increasing the number with a 0 at the end of the current limit, for example 65492030, should be enough. To set this value for a live system, use this command as root, sudo will probably not work:
~$ sudo -s
[sudo] password for user:
~# cat /proc/sys/fs/file-max
~# echo 65492030 >> /proc/sys/fs/file-max
~# cat /proc/sys/fs/file-max
To make this change permanent, edit or add the following line in the
fs.file-max = 65492030
If you have increased this value to the max or have increased the default number more than once, please submit a support ticket.