Fix Out of Memory Errors

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

On this page:

Updating JVM options when running Fisheye / Crucible on Windows

If the Windows service is installed
  1. The Apache procrun monitor application is used for editing the service and it is installed automatically. You can usually open it by going to Windows Start Menu > All Programs > Fisheye > Configure Fisheye.
    If this path cannot be found in the Windows Start Menu:
    1. Open the Windows services list and open the Properties panel for the Fisheye service in order to copy the Service Name to clipboard (it's usually named Atlassian Fisheye).
    2. In a Command Prompt window, navigate to <Fisheye Installation Directory>/bin and run a command like this:

      prunmgr.exe "//ES//SERVICE_NAME"

      Where SERVICE_NAME  needs to be replaced by what has been copied to clipboard, for example:

      prunmgr.exe "//ES//Atlassian Fisheye"
  2. The tool will be opened:
  3.  Go to Java tab in order to update the JVM options as needed:

    1. The Initial memory pool setting configures the -Xms JVM option.
    2. The Maximum memory pool setting configures the -Xmx JVM option.
    3. Any other JVM option recommended in the troubleshooting section of this document needs to be configured in the Java Options list, each JVM option in a new line.
      Please remember:
      1. If it's a Fisheye environment variable (such as FISHEYE_INST, FISHEYE_OPTS, etc), add -D at the beginning of the line and type the option name in lowercase. The option value can be camel cased. Example:

        -Dfisheye.inst=C:\Atlassian\fecru_inst
      2. If it's a regular JVM option there is no need to add -D  at the beginning. Example:

        -XX:+HeapDumpOnOutOfMemoryError
  4. Apply changes and restart the Windows service so that the changes are picked by the JVM.
  5. You can confirm that the settings were updated correctly using the Fisheye web interface by navigating to Administration > System Settings > System Info and looking for the JVM input arguments label.
If the Windows service is not installed
  1. Modify JVM options as needed in the FISHEYE_OPTS environment variable (see the page on Environment variables).
  2. Fisheye's own variables such as FISHEYE_ARGS are declared separately as environment variable.
  3. To check if the environment variable is set and what its value currently is, use the echo command. For example:

    echo %FISHEYE_INST%
    echo %FISHEYE_OPTS%
  4. Restart Fisheye / Crucible so that the changes are picked by the JVM.
  5. You can confirm that the settings were updated correctly using the Fisheye web interface by navigating to Administration > System Settings > System Info and looking for the JVM input arguments label.

Updating JVM options when running Fisheye / Crucible on Linux or MacOS

Click here to see instructions

At Installing Fisheye on Linux and Mac we recommend setting environment variables in the /etc/environment file on Linux distributions and in the ~/.profile file for the current user on MacOS.

Environment variables can be set in multiple different bash configuration files though. In most distributions, when you start a new session the environment variables are read from the following files:

  • /etc/environment: Use this file to set up system-wide environment variables. Variables in this file are set in the following format:

    FISHEYE_INST="/absolute/path/to/<Fisheye data directory>"
    FISHEYE_OPTS="-Xms2g -Xmx2g"
  • /etc/profile: Variables set in this file are loaded whenever a bash login shell is entered. When declaring environment variables in this file you need to use the export command:

    export FISHEYE_INST="/absolute/path/to/<Fisheye data directory>"
    export FISHEYE_OPTS="-Xms2g -Xmx2g"
  • ~/.bashrc, ~/.bash_profile or ~/.profile: Per-user shell specific configuration files. When declaring environment variables in this file you need to use the export command:

    export FISHEYE_INST="/absolute/path/to/<Fisheye data directory>"
    export FISHEYE_OPTS="-Xms2g -Xmx2g"
  • To load the new environment variables into the current shell session use the source command. For example:

    source ~/.bashrc
  • To check if the environment variable is set and what its value currently is, use the echo command. For example:

    echo $FISHEYE_INST
    echo $FISHEYE_OPTS
  • You can confirm that the settings were updated correctly using the Fisheye web interface by navigating to Administration > System Settings > System Info and looking for the JVM input arguments label.

トラブルシューティング

There are a number of different Out Of Memory Errors that the JVM can encounter. Solving these errors might require that you update the default memory settings by following the steps from the previous sections of this article, according to Fisheye's host operating system.

The most common types of Out Of Memory Errors are listed below:

OutOfMemoryError: Java Heap Space

  • Meaning: The JVM had to allocate more memory than it is currently allowed to.
  • Default value: 1024 MB / 1 GB
  • Known scenarios: 
    • During the initial repository scan period
    • While syncing a large number of users from an external user directory
    • While a large backup is being performed
    • When working on large reviews
    • Installed add-ons
  • How to fix it: This usually indicates you need to increase the amount of memory the JVM can allocate (also known as memory heap) to Fisheye, however doing so may mask and / or postpone the real issue. In addition to that, there are many things to consider when increasing the available memory to your application. For example, the amount of memory being allowed cannot be greater than the RAM configured on the server and also needs to take into consideration the memory required by all the other applications installed on the system and the amount of memory required by the operating system itself. It may be needed to increase the RAM on the server before actually increasing the amount of memory the JVM used by Fisheye can allocate.
    When needed, though, to increase the amount of memory the JVM can allocate you need to:
    • If Fisheye runs on Windows and the Windows service for Fisheye exists, set / increase the Maximum memory pool field in the Apache procrun monitor application (see instructions in the section above).
    • If Fisheye runs on Windows and the Windows service for Fisheye does not exist, set / increase the -Xmx option in the FISHEYE_OPTS environment variable (see instructions in the section above).
    • If Fisheye runs on Linux or Mac, set / increase the -Xmx option in the FISHEYE_OPTS environment variable (see instructions in the section above).
    • Restart Fisheye afterwards.

OutOfMemoryError: unable to create new native thread

  • Meaning: This error occurs when the operating system is unable to create new threads.
  • Default value: not set by default.
  • Known scenarios:
    • The JVM heap may be taking up the available RAM on the system. Big heaps take away from the space that can be allocated for the stack of a new thread.
    • The OS limits for the maximum number of threads are too low
  • How to fix it:
    • Option 1: Reduce the heap: 

      The size of the stack per thread (-Xss) can also contribute to this problem. The stack size can reduce the number of threads that can be created. 
      The default value of -Xss depends on the platform: 
      - On Windows the default value depends on virtual memory.

      - On Linux / MacOS the default value is 1024 KB.

      To fix this problem, you should reduce the amount of memory the JVM can allocate (-Xmx) and also reduce the size of the JVM stack per thread (-Xss).
      • If Fisheye runs on Windows and the Windows service for Fisheye exists, open the Apache procrun monitor application and:
        • Set / decrease the Maximum memory pool field
        • Set the -Xss option in the Java Options list to, for example, -Xss512k.
        • See instructions in the section above.
      • If Fisheye runs on Windows and the Windows service for Fisheye does not exist, or if Fisheye runs on Linux or MacOS:
        • Set / decrease the -Xmx option in the FISHEYE_OPTS environment variable
        • Set the -Xss option to -Xss512k in the FISHEYE_OPTS environment variable
        • See instructions in the section above for each operating system.
      • Restart Fisheye afterwards.
    • Option 2: Increase the number of allowed threads:
      In Linux / MacOS systems the maximum number of allowed threads can be checked by executing ulimit -u as the user running Fisheye/Crucible.
      This limit can be increased by executing ulimit -u 2048 for example, which will set the maximum number of threads to 2048.
      Restart Fisheye afterwards.

OutOfMemoryError: GC overhead limit exceeded

  • Meaning: This error indicates the JVM took too long to free up memory during its garbage collection process.
  • Default value: not set by default.
  • Known scenarios: The Oracle Java documentation states that this error is reported "if more than 98% of the total time is spent in garbage collection and less than 2% of the heap is recovered", and that "this feature is designed to prevent applications from running for an extended period of time while making little or no progress because the heap is too small". This may happen if the JVM starts to use the swapped memory for its heap. This will cause the JVM to take longer than normal to perform normal GC operations which will eventually cause a timeout to occur.
  • How to fix it: Even though this feature can be disabled by adding the -XX:-UseGCOverheadLimit JVM option to stop this kind of error to be written in the Fisheye log files, it is not a good idea to do so. To overcome this issue, you actually need to make sure that all processes can't allocate more memory than the RAM available on the system. In practice this is impossible to calculate that for all processes so at a minimum you should make sure that the JVM does not have a total maximum memory allocation than your normally available system memory.
    To analyze the garbage collection activity and to explore the contents the application's memory, the JVM options below should be used. Please check in the sections above how to add these options according to your use case:

    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=<absolute_path_to_file>
    -XX:+PrintGCTimeStamps
    -XX:+PrintGCDateStamps 
    -verbose:gc
    -Xloggc:<absolute_path_to>/atlassian-fecru-gc-$(date +%Y-%m-%d_%H%M).log 
    -XX:+PrintGCDetails

    Restart your Fisheye instance after adding these options above and enable debug logging via Fisheye web interface by going to Administration > Global Settings > Server > Debug Logging and clicking the Turn Debugging ON button.

    The atlassian-fecru-gc-<timestamp>.log file will be created
     automatically on restart, however the heap dump file named like java_pid<process PID here>.hprof (e.g. java_pid12345.hprof) that is the size of the heap will only be created on the next OutOfMemoryError.

    If you open a support request with the following items the Atlassian Support team will be able to investigate this problem further:
    - The atlassian-fecru-gc-<timestamp>.log
    - The java_pid<process PID here>.hprof
    - The support zip with debug logging enabled before the error was reported.

    The garbage collection settings (excluding the -Xloggc one) can be enabled at runtime as explained on the How to change JVM arguments at runtime to avoid application restart guide.

java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

  • Meaning: The native objects do not have enough memory to use.
  • Known scenarios: This usually happens when you have allocated too much memory to your heap, reducing the amount available for native objects. See this article.
  • How to fix it: Reduce the amount of heap memory you have allocated. For example, if you currently allow the JVM to allocate up to 4096m of memory, you should consider reducing this to 3072m or even 2048m. 

Additional reading

The following documentation provides additional clarifications on adjusting limits and performance settings in Fisheye:


Last modified on Mar 17, 2022

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.