このページの内容

Out Of Memory Errors (OOMEs)

There are a number of different memory errors that the JVM can encounter – the most common ones are listed below. Solving these errors might require that you update the default memory settings which is accomplished by modifying the FISHEYE_OPTS environment variable (see the page on Environment variables) and then restarting the application. You can confirm that the settings were updated correctly by confirming that the updated parameters and values appear in the UI by navigating to Administration > Sys Info/Support > System Info and looking at "JVM Input Arguments".

OutOfMemoryError: Java Heap Space

In some cases you need to increase the amount of memory allocated to Fisheye during the initial repository scan period but once this has completed you can reduce back down to the default value. There are many variables to consider when increasing the available memory to your application (total system memory, other applications' memory requirements, etc.) but this can be accomplished using the Xmx parameter:

FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m"

After having set the FISHEYE_OPTS and restarting your server, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.

OutOfMemoryError: PermGen space, or Permanent Generation Size

If you get the error message: java.lang.OutOfMemoryError: PermGen space this means that you have exceeded Java's fixed 64MB block for loading class files. You will need to add the argument -XX:MaxPermSize=256m to FISHEYE_OPTS, in addition to any argument you use to set the heap size.

FISHEYE_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m"

After having set the FISHEYE_OPTS and restarting Fisheye/Crucible, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.

OutOfMemoryError: unable to create new native thread

This error occurs when the operating system is unable to create new threads. This can be due to the JVM Heap taking up the available RAM or that the OS limits for the maximum number of threads are too low.

Big heaps take away from the space that can be allocated for the stack of a new thread

Option 1: Reducing Heap

For Linux the maximum heap size of the 32bit JVM cannot be greater than 2GB. If you only have 2GB RAM in your server, it is not recommended to set the Max size of the JVM that high.
The size of the stack per thread can also contribute to this problem. The stack size can reduce the number of threads that can be created.

To fix this problem, you should reduce the size of your JVM Heap (Xmx) and also the size of the JVM stack per thread (Xss) by adding the following parameters to FISHEYE_OPTS, e.g.:

FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Xss512k"

After having set the FISHEYE_OPTS and restarting your server, go to Administration > Sys Info/Support > System Info, and check your JVM Input Arguments to ensure that your server is picking up your FISHEYE_OPTS as expected.

Option 2: Increase the number of allowed threads

The maximum number of allowed threads can be checked by executing ulimit -u as the user running Fisheye/Crucible. This limit can be increased by executing ulimit -u 2048 for example, which will set the maximum number of threads to 2048.

OutOfMemoryError: GC overhead limit exceeded

This error indicates that the JVM took too long to free up memory during its garbage collection process. This error can be thrown from the Parallel or Concurrent collectors. From the Oracle Java documentation, the error is reported

 if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered ...This feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small.

This kind of OutOfMemoryError can be caused if your java process is starting to use swapped memory for its heap. This will cause the JVM to take longer than normal to perform normal GC operations which will eventually cause a timeout to occur and cause this error.

To overcome this issue, you need to make sure that all processes can't allocate more memory than there is system memory. In practice this is impossible to do for all processes so at a minimum you should make sure that all your application JVMs do not have a total maximum memory allocation than your normally available system memory.

To analyze the garbage collection activity and to explore the contents the application's memory the following JVM options should be added to FISHEYE_OPTS, e.g.:

FISHEYE_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m -Xss512k -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<fullpath_to_file> -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -verbose:gc -Xloggc:<fullpath_to_file>/atlassian-fecru-gc-$(date +%Y-%m-%d_%H%M).log

 -XX:+PrintGCDetails"

The file gc.log will be created automatically on restart and but only on the next OutOfMemoryError will a file (java_pid<process PID here>.hprof, e.g. java_pid12345.hprof) be created that is the size of the heap. If you open a support request with the following items the Atlassian Support team will be able to investigate:

• gc.log
• java_pidXXXX.hprof
• Support Zip

Please refer to this guide for more information.

The garbage collection settings (excluding the -Xloggc one) can be enabled at runtime as explained on the How to change JVM arguments at runtime to avoid application restart guide.

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

Essentially the native objects does not have enough memory to use. This is usually because you have allocated too much memory to your heap reducing the amount available for native objects. See this article.

The solution is to reduce the amount of heap memory you have allocated. For example if you have set -Xmx4096, you should consider reducing this to -Xmx2048m.

Remember if you are using a 32bit JVM you cannot allocate more than -Xmx2048m for linux (and even less for windows). Using a 64 bit JVM resolves this problem.

Read the Tuning Fisheye page for more detail on adjusting resource limits and performance settings in Fisheye.