Bamboo Server Process Dies Unexpectedly Due to Linux OOM-Killer

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: サーバーと Data Center のみ。この記事は、サーバーおよび Data Center プラットフォームのアトラシアン製品にのみ適用されます。

症状

  • Bamboo Server is installed on a Linux host.
  • The entire Bamboo Server process suddenly terminates without warning. That is to say, the Process ID (PID) is gone.
  • ブラウザには汎用的な "接続できませんでした" あるいは類似のエラーが表示され、web ページに到達できない旨が示される。
  • Nothing out of the ordinary appears in the Bamboo Server application logs (<bamboo-home>/logs/atlassian-bamboo.log), since the application was terminated without properly shutting down
  • Similarly, Tomcat logs (<bamboo-install>/logs/catalina.out) also do not show errors

原因と診断

Bamboo Server does not terminate its own PID unless a stop-bamboo.sh is executed (which writes shutdown messages to the log). An unexpected PID termination clearly indicates the work of an external entity. Some examples include:

  • kill -9 (user-triggered or via script) 
  • Linux OOM-Killer

This KB article focuses on the Linux OOM-Killer, which is a feature on some Linux installations that will sacrifice processes to free up memory if the operating system experiences memory exhaustion for its own operations. Please note that this is different from Bamboo Server running out of memory. In this case, the OS itself is in danger of running out of memory and thus starts terminating processes to avoid it.

On the host machine, look in the /var/log/ directory for the syslog or messages, and locate the timestamps spanning the approximate time when the PID was terminated. If you see entries similar to the following, then you know the process was a victim of the OOM-Killer:

Apr  8 11:29:48 Bamboo01 kernel: Out of memory: Kill process 1386 (java) score 358 or sacrifice
Apr  8 11:29:48 Bamboo01 kernel: Killed process 5388, UID 4048, (java) total-vm:1331564kB, anon
Apr  8 11:29:51 Bamboo01 kernel: java invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0,

Use can use dmesg and search for lines around the killed process.

dmesg -T | grep -C 5 -i “killed process”

回避策

The presented information is a workaround used during an emergency. This solution is not expected to cover all Out of Memory issues. Further diligence on each operating system's characteristics and proper capacity planning of the Linux Server may be necessary to find the root cause of the memory issues. The following recommendations do not consider any other applications that are running concurrently on the same server. It is the Operating system administrator to understand and plan the right amount of memory that will be available.

We have considered using a file on your Linux root disk as a Swap location. As a best practice, you can also use a disk partition or logical volume.

The way to mitigate this type of issue is to Add Swap Space to the system. Having a Swap location will keep Bamboo from crashing while working on a permanent resolution (Adding RAM). This can be done live without having to stop Linux or Bamboo. This will have performance impacts on Bamboo so this should only be used to prevent a full outage. 

  • Decide where on the disk there is available space.  This example will put the swap space in the root user's home directory

  • Create a file on the local disk.  This is an example of a 2G swap file:

    # dd if=/dev/zero of=/root/myswapfile bs=1M count=2048
  • Change the permission of the swap file so that only the root user can access it

    # chmod 600 /root/myswapfile
  • Make and enable this file as a swap file using the mkswap command

    # mkswap /root/myswapfile
    # swapon /root/myswapfile
  • Verify whether the newly created swap area is available for your use

    # swapon -s

ソリューション

OOM-Killer が原因である場合は次のようなソリューションが考えられます。

  • Increase the amount of memory available on the host machine itself.

  • Decrease the amount of memory allocated to Bamboo Server or competing processes on the machine: Configuring your System Properties
  • Profile the memory usage during periods of high usage and understand if there are other processes increasing the footprint. For example, running the below ps command will help identify the top 10 memory consumers at the time it is run:

    $ ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -10
  • Disable the OOM-Killer (not recommended).

Additional info on how the OOM-Killer operates, please see: http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/

Last modified on Mar 28, 2022

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.