Fix Out of Memory errors
このページの内容
Updating JVM options when running the application as a Windows service
For Fisheye/Crucible 3.4 and later, 32-bit and 64-bit Windows installers are available. Each installer sets up the service wrapper, adds Fisheye as a Windows service, and starts the service, automatically. Prior to the introduction of the installers, running the application as a service required the use of the Java Service Wrapper. See Running Fisheye as a Windows service for more information.
Installer - versions: 3.4.3+
The tool to edit the service is installed automatically and you can find it here: Windows Start Menu > All Programs > FishEye > Configure FishEye
. Modify the JVM options by selecting the tab labeled Java and then restart the service so the settings take effect.
Installer - versions: 3.4 to 3.4.3
You must download the tool mentioned above to edit the service. Download it here:
http://www.apache.org/dist/commons/daemon/binaries/windows/commons-daemon-1.0.15-bin-windows.zip
To use it, you need to rename "prunmgr.exe" from the zip file to "Atlassian Fisheye.exe" or "Atlassian Crucible.exe" depending on which product you initially installed.
Java Service Wrapper - versions: all (not recommended for new installations)
Updating the JVM options with this setup requires updating the wrapper configuration file, wrapper.conf
. Refer to Running Fisheye as a Windows Service under Setting Fisheye environment variables for Windows Services for instructions.
Out Of Memory Errors (OOMEs)
There are a number of different memory errors that the JVM can encounter – the most common ones are listed below. Solving these errors might require that you update the default memory settings which is accomplished by modifying the FISHEYE_OPTS
environment variable (see the page on Environment variables) and then restarting the application. You can confirm that the settings were updated correctly by confirming that the updated parameters and values appear in the UI by navigating to Administration > Sys Info/Support > System Info
and looking at "JVM Input Arguments".
OutOfMemoryError: Java Heap Space
In some cases you need to increase the amount of memory allocated to Fisheye during the initial repository scan period but once this has completed you can reduce back down to the default value. There are many variables to consider when increasing the available memory to your application (total system memory, other applications' memory requirements, etc.) but this can be accomplished using the Xmx
parameter:
FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m"
After having set the FISHEYE_OPTS and restarting your server, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.
OutOfMemoryError: PermGen space, or Permanent Generation Size
If you get the error message: java.lang.OutOfMemoryError: PermGen space
this means that you have exceeded Java's fixed 64MB block for loading class files. You will need to add the argument -XX:MaxPermSize=256m
to FISHEYE_OPTS, in addition to any argument you use to set the heap size.
FISHEYE_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m"
After having set the FISHEYE_OPTS and restarting Fisheye/Crucible, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.
OutOfMemoryError: unable to create new native thread
This error occurs when the operating system is unable to create new threads. This can be due to the JVM Heap taking up the available RAM or that the OS limits for the maximum number of threads are too low.
Big heaps take away from the space that can be allocated for the stack of a new thread
Option 1: Reducing Heap
For Linux the maximum heap size of the 32bit JVM cannot be greater than 2GB. If you only have 2GB RAM in your server, it is not recommended to set the Max size of the JVM that high.
The size of the stack per thread can also contribute to this problem. The stack size can reduce the number of threads that can be created.
To fix this problem, you should reduce the size of your JVM Heap (Xmx
) and also the size of the JVM stack per thread (Xss
) by adding the following parameters to FISHEYE_OPTS, e.g.:
FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Xss512k"
After having set the FISHEYE_OPTS and restarting your server, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.
Option 2: Increase the number of allowed threads
The maximum number of allowed threads can be checked by executing ulimit -u
as the user running Fisheye/Crucible. This limit can be increased by executing ulimit -u 2048
for example, which will set the maximum number of threads to 2048.
OutOfMemoryError: GC overhead limit exceeded
This error indicates that the JVM took too long to free up memory during its garbage collection process. This error can be thrown from the Parallel or Concurrent collectors. From the Oracle Java documentation, the error is reported
if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered ...This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.
This kind of OutOfMemoryError can be caused if your java process is starting to use swapped memory for its heap. This will cause the JVM to take longer than normal to perform normal GC operations which will eventually cause a timeout to occur and cause this error.
To overcome this issue, you need to make sure that all processes can't allocate more memory than there is system memory. In practice this is impossible to do for all processes so at a minimum you should make sure that all your application JVMs do not have a total maximum memory allocation than your normally available system memory.
To analyze the garbage collection activity and to explore the contents the application's memory the following JVM options should be added to FISHEYE_OPTS, e.g.:
FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Xss512k -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<fullpath_to_file> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -verbose:gc -Xloggc:<fullpath_to_file>/atlassian-fecru-gc-$(date +%Y-%m-%d_%H%M).log
-XX:+PrintGCDetails"
The file gc.log
will be created automatically on restart and but only on the next OutOfMemoryError will a file (java_pid<process PID here>.hprof, e.g. java_pid12345.hprof) be created that is the size of the heap. If you open a support request with the following items the Atlassian Support team will be able to investigate:
gc.log
java_pidXXXX.hprof
- Support Zip
Please refer to this guide for more information.
The garbage collection settings (excluding the -Xloggc
one) can be enabled at runtime as explained on the How to change JVM arguments at runtime to avoid application restart guide.
java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
Essentially the native objects does not have enough memory to use. This is usually because you have allocated too much memory to your heap reducing the amount available for native objects. See this article.
The solution is to reduce the amount of heap memory you have allocated. For example if you have set -Xmx4096, you should consider reducing this to -Xmx2048m.
Remember if you are using a 32bit JVM you cannot allocate more than -Xmx2048m for linux (and even less for windows). Using a 64 bit JVM resolves this problem.
Read the Tuning Fisheye page for more detail on adjusting resource limits and performance settings in Fisheye.