Bitbucket Process Dies Unexpectedly Due to Linux OOM-Killer

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問


プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

問題

Bitbucket does not terminate its own process unless a stop-bitbucket.sh is executed which writes shutdown messages to the log. An unexpected process termination clearly indicates the work of an external entity. Some examples include:

  •  kill -9 (user triggered or via script) 
  • Linux OOM-Killer

This KB article focuses on the Linux OOM-Killer, which is a feature on some Linux installations that will sacrifice processes to free up memory if the operating system experiences memory exhaustion for its own operations.  Please note that this is different from Bitbucket running out of memory. In this case, the OS itself is in danger of running out of memory and thus starts terminating processes to avoid it.  If Bitbucket is shutdown correctly, this is an example log from a Bitbucket Server.  If this log exists, Bitbucket was shutdown most likely using the stop-bitbucket.sh command and this article does not apply. 

2019-05-28 18:21:39,435 INFO  [Thread-4]  c.a.b.i.boot.log.BuildInfoLogger Bitbucket 5.16.0 has shut down

診断

環境

  • Bitbucket is installed on a Linux host.
  • The entire Bitbucket process suddenly terminates without warning. That is to say, the process ID (pid) is no longer running and there is no log message as the example above.
  • The browser shows a generic "cannot connect" or similar error, indicating that it is not able to reach the webpage.
  • Nothing out of the ordinary appears in the Bitbucket application logs ($BITBUCKET_HOME/logs/atlassian-bitbucket.log), since the application was terminated without properly shutting down.

Diagnostic Steps

  • /var/log/ directory for the syslog or messages, and locate the timestamps spanning the approximate time when the process was terminated.  If you see entries similar to the following, then you know the process was a victim of the OOM-Killer:

    • May 28 11:29:48 Bitbucket01 kernel: Out of memory: Kill process 1386 (java) score 358 or sacrifice
      May 28 11:29:48 Bitbucket01 kernel: Killed process 5388, UID 4048, (java) total-vm:1331564kB, anon
      May 28 11:29:51 Bitbucket01 kernel: java invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0,
  • Use dmesg and search for lines around Killed process.

    • dmesg -T | grep -C 5 -i “killed process”

原因

The linux system experiences memory exhaustion for its own operations.  In an attempt to save the system from crashing, Linux will kill process to free memory to continue running.

回避策

The way to mitigate this type issue is to Add Swap Space to the system.  This will keep Bitbucket from crashing while working on the resolution.  This will have performance impacts on Bitbucket so this should only be used to prevent a full outage for Bitbucket. 

  • Decide where on the disk there is available space.  This example will put the swap space in the root user's home directory.

  • Create a file on the local disk.  This is an example of 2G swap file: 

    dd if=/dev/zero of=/root/myswapfile bs=1M count=2048
  • Change the permission of the swap file so that only root can access it. 

    chmod 600 /root/myswapfile
  • Make and enable this file as a swap file using mkswap command.

    mkswap /root/myswapfile
    swapon /root/myswapfile
  • Verify whether the newly created swap area is available for your use. 

    swapon -s

ソリューション

OOM-Killer が原因である場合は次のようなソリューションが考えられます。

  • Increase the amount of memory available on the host machine itself.

  • Decrease the amount of memory allocated to Bitbucket or competing processes on the machine.
  • Migrate other applications to another system.

(info) Additional info on how the OOM-Killer operates, please see: http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/



説明 Bitbucket becomes unresponsive. Application just stops.
製品Bitbucket, Bitbucket Server, Bitbucket Data Center
最終更新日 2019 年 5 月 29 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.