Stash Process Dies Unexpectedly Due to Linux OOM-Killer
症状
- Stash is installed on a Linux host.
- The entire Stash process suddenly terminates without warning. That is to say, the process ID (pid) is gone.
- ブラウザには汎用的な "接続できませんでした" あるいは類似のエラーが表示され、web ページに到達できない旨が示される。
- Nothing out of the ordinary appears in the Stash application logs (
$STASH_HOME/logs/atlassian-stash.log), since the application was terminated without properly shutting down - Similarly, Tomcat logs (
<Stash installation directory>/logs/catalina.out) also do not show errors
原因と診断
Stash does not terminate its own pid unless a stop-stash.sh is executed (which writes shutdown messages to the log). An unexpected pid termination clearly indicates the work of an external entity. Some examples include:
-
kill -9(user triggered or via script) - Linux OOM-Killer
This KB article focuses on the Linux OOM-Killer, which is a feature on some Linux installations that will sacrifice processes to free up memory if the operating system experiences memory exhaustion for its own operations. Please note that this is different from Stash running out of memory. In this case, the OS itself is in danger of running out of memory and thus starts terminating processes to avoid it.
On the host machine, look in the /var/log/ directory for the syslog or messages, and locate the timestamps spanning the approximate time when the pid was terminated. If you see entries similar to the following, then you know the process was a victim of the OOM-Killer:
Apr 8 11:29:48 stash01 kernel: Out of memory: Kill process 1386 (java) score 358 or sacrifice
Apr 8 11:29:48 stash01 kernel: Killed process 5388, UID 4048, (java) total-vm:1331564kB, anon
Apr 8 11:29:51 stash01 kernel: java invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0,
ソリューション
OOM-Killer が原因である場合は次のようなソリューションが考えられます。
Increase the amount of memory available on the host machine itself.
- Decrease the amount of memory allocated to Stash or competing processes on the machine.
- Disable the OOM-Killer (not recommended).
OOM-Killer の処理の詳細については次のページをご確認ください。http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/