Bamboo is encountering Too many open files errors after recent Java (JDK) update due to Socket leak
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bamboo is encountering (Too many open files) errors after recent Java (JDK) update. At this point, it's known to affect JDK 1.8.0_271 and 1.8.0_281. This can cause operations to error out frequently, render the application unusable or cause it to crash. Analysis of the lsof output for the Bamboo process shows that a disproportionate number of open files are related to TCP sockets that are never released / leaking.
Please note, there are other causes for this such as having ulimits set too low for what is required or a bug that was fixed in Bamboo 6.9.2. This is discussed more indepth in the below knowledge-base article and bug reports
Environment
JDK 1.8.0_271 < b37
JDK 1.8.0_281 < b31
JDK 1.8.0_275 does not appear to be affected.
Diagnosis
$BAMBOO_HOME/logs/atlassian-bamboo.log contains many errors about Too many open files. Example:
1
Caused by: java.lang.IllegalStateException: java.io.FileNotFoundException: /opt/atlassian/bamboo/atlassian-bamboo/WEB-INF/lib/... (Too many open files)Check the lsof output for the Bamboo Process ID (PID):
1
2
BAMBOO_PID=$(ps aux | grep bamboo | grep Bootstrap | awk '{print $2}')
lsof -p $BAMBOO_PIDIt shows a high number of open file descriptors are unbound sockets. Example:
1
2
3
4
5
6
7
8
9
10
11
12
13
...
java 50129 bamboo 253u sock 0,9 0t0 3282247 protocol: TCPv6
java 50129 bamboo 254u sock 0,9 0t0 3282248 protocol: TCPv6
java 50129 bamboo 255u sock 0,9 0t0 3282249 protocol: TCPv6
java 50129 bamboo 256u sock 0,9 0t0 3282250 protocol: TCPv6
java 50129 bamboo 257u sock 0,9 0t0 3282251 protocol: TCPv6
java 50129 bamboo 258u sock 0,9 0t0 3282252 protocol: TCPv6
java 50129 bamboo 259u sock 0,9 0t0 3282253 protocol: TCPv6
java 50129 bamboo 260u sock 0,9 0t0 3282254 protocol: TCPv6
java 50129 bamboo 261u sock 0,9 0t0 3282255 protocol: TCPv6
java 50129 bamboo 262u sock 0,9 0t0 3282256 protocol: TCPv6
java 50129 bamboo 263u sock 0,9 0t0 3282257 protocol: TCPv6
...⚠️ This count will increase overtime when leaking sockets:
1
2
lsof -p $BAMBOO_PID | grep sock | grep protocol | wc -l
1144(If Bamboo is currently running) Check the Java version your Bamboo is using:
Bamboo Administration >> System Information >> Java / JVM Information: JVM version
If Bamboo is not running, check the Java version your Bamboo typically uses:
1
$JAVA_HOME/jre/bin/java -versionAffected versions are listed in the Environment section above.
Cause
Solution
Upgrade the JDK that your Bamboo installation uses to JDK 1.8.0_282 or more recent. AdoptOpenJDK provides distributions of this which you can get from the below location:
Ensure that the $JAVA_HOME environment variable is updated to this new JDK for the user and/or service that launches Bamboo.
Alternatively, the first affected version is claimed to be 271, so downgrading to Java 8 pre 271 may also be a viable option – but we recommend being on the later version with a full fix.
Was this helpful?