Bitbucket Data Center node hangs on startup - Initializing spring context
プラットフォームについて: Data Center のみ - この記事は、Data Center プラットフォームのアトラシアン製品にのみ適用されます。
この KB は Data Center バージョンの製品用に作成されています。Data Center 固有ではない機能の Data Center KB は、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
要約
Bitbucket Data Center node(s) do not start up. Startup stuck on the "Initializing spring context" step. Neither atlassian-bitbucket.log
, nor catalina.out
show any particular errors.
As of Bitbucket Server 5.x, catalina.out
will no longer exist. It will be written to atlassian-bitbucket.log
instead.
環境
- Bitbucket Data Center using shared NFS storage.
- File creation/modification on the NFS storage works as expected.
診断
- While startup is stuck, take thread dumps to identify why the startup hangs.
If the following thread is seen in long running threads, you may be encountering the problem(s) outlined below:
"spring-startup" #18 daemon prio=5 tid=0x00007f4511e75000 nid=0x5afe runnable [0x00007f45a13dd000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90) at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115) at com.atlassian.stash.internal.home.HomeLock.acquireLock(HomeLock.java:108) at com.atlassian.stash.internal.home.HomeLock.lock(HomeLock.java:94) at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1706) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1645) ...
You can use a C program to validate NFS locking ability. You will simply need to put the outlined C code into a file then compile it. When done, you can move it to the NFS share on the client (application node) and run it. It should finish in less than a second and create
testfile
for testing file locking. If this fails, locking almost certainly doesn't work.
原因
The above thread indicates that Bitbucket fails to acqurie a lock on the shared storage (NFS). The causes may be - but not limited to the - following:
- NFS daemons (lockd, nfsd, mountd, etc) not running on either the client or server side
- NFS daemons are hung or have problems communicating
- Permission problems on the NFS share
- Firewall blocking connection for either of the above listed daemons. Check which ports the required daemons run by executing
portmap -p
on the NFS server and make sure that the relevant TCP and UDP ports are accepting connections.
ソリューション
Make sure that all the NFS daemons are running and accessible from the client perspective. As a troubleshooting step, try disabling firewall on both the NFS server and client(s), test NFS locking manually and ensure it works as expected. Please note that this error is very likely caused by an environmental issue, rather than Bitbucket Data Center not operating as expected.
参考
- Running NFS behind a firewall (Redhat)
- Running NFS behind a firewall (Fedora)
- Firewalling a Linux NFS server with iptables
- C program to validate NFS locking ability
- - BSERV-8969Getting issue details... STATUS