Bitbucket Data Center node hangs on startup - Initializing spring context

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Data Center のみ - この記事は、Data Center プラットフォームのアトラシアン製品にのみ適用されます。

この KB は Data Center バージョンの製品用に作成されています。Data Center 固有ではない機能の Data Center KB は、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。

*Fisheye および Crucible は除く

要約

Bitbucket Data Center node(s) do not start up. Startup stuck on the "Initializing spring context" step. Neither atlassian-bitbucket.log, nor catalina.out show any particular errors.

(info) As of Bitbucket Server 5.x, catalina.out will no longer exist. It will be written to atlassian-bitbucket.log instead. 

環境

  • Bitbucket Data Center using shared NFS storage.
  • File creation/modification on the NFS storage works as expected.

診断

  • While startup is stuck, take thread dumps to identify why the startup hangs.
  • If the following thread is seen in long running threads, you may be encountering the problem(s) outlined below: 

    "spring-startup" #18 daemon prio=5 tid=0x00007f4511e75000 nid=0x5afe runnable [0x00007f45a13dd000]
       java.lang.Thread.State: RUNNABLE
    	at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)
    	at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90)
    	at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115)
    	at com.atlassian.stash.internal.home.HomeLock.acquireLock(HomeLock.java:108)
    	at com.atlassian.stash.internal.home.HomeLock.lock(HomeLock.java:94)
    	at com.atlassian.stash.internal.home.HomeLockAcquirer.lock(HomeLockAcquirer.java:58)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1706)
    	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1645)
    
    ...
  • You can use a C program to validate NFS locking ability. You will simply need to put the outlined C code into a file then compile it. When done, you can move it to the NFS share on the client (application node) and run it. It should finish in less than a second and create testfile for testing file locking. If this fails, locking almost certainly doesn't work.

    • See the practical example

      Put the following contents into a file nfs_lock.c then execute either make nfs_lock or gcc -o nfs_lock nfs_lock.c, then copy and run the command nfs_lock from the NFS share on the client (application node):

      #include <stdio.h> 
      #include <fcntl.h> 
      main() { 
          int fd , ret; 
          struct flock strlock; 
          fd = open("testfile", O_CREAT | O_RDWR , 0666 ); 
          if (fd == -1){ 
              printf("FATAL ERROR: Could not open file\n"); 
              exit(-1); 
          } 
       
      /* Set a write lock on the first 1024 bytes of the file. 
         It doesn't matter that they don't actually exist yet. */ 
       
          strlock.l_type = F_WRLCK; 
          strlock.l_whence = 0; 
          strlock.l_start = 0L; 
          strlock.l_len = 1024L; 
          ret = fcntl (fd, F_SETLK , &strlock); 
          if (ret == -1){ 
              printf("FATAL ERROR: Could not lock file\n"); 
              exit(-1); 
          } 
          ret = close( fd ); 
          if (ret == -1){ 
              printf("FATAL ERROR: Could not close file\n"); 
              exit(-1); 
          } 
      } 

      Source: http://community.microfocus.com/microfocus/cobol/net_express__server_express/w/knowledge_base/6215.c-program-to-validate-nfs-locking-ability.aspx

原因

The above thread indicates that Bitbucket fails to acqurie a lock on the shared storage (NFS). The causes may be - but not limited to the - following:

  • NFS daemons (lockd, nfsd, mountd, etc) not running on either the client or server side
  • NFS daemons are hung or have problems communicating
  • Permission problems on the NFS share
  • Firewall blocking connection for either of the above listed daemons. Check which ports the required daemons run by executing portmap -p on the NFS server and make sure that the relevant TCP and UDP ports are accepting connections.

ソリューション

Make sure that all the NFS daemons are running and accessible from the client perspective. As a troubleshooting step, try disabling firewall on both the NFS server and client(s), test NFS locking manually and ensure it works as expected. Please note that this error is very likely caused by an environmental issue, rather than Bitbucket Data Center not operating as expected.

参考


Last modified on Mar 28, 2024

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.