SSH Fail Intermittently due to upload-pack: Resource temporarily unavailable

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server と Data Center のみ - この記事は、サーバーおよびデータセンター プラットフォームのアトラシアン製品にのみ適用されます。

問題

  • SSH git clone fail intermittently while HTTP(S) works as per normal.
  • This is noticed after an upgrade to Bitbucket Server 3.6.0 - 3.10.x

catalina.log に次のログが記録されます。

30-Jun-2015 04:38:43.453 SEVERE [ajp-nio-127.0.0.1-8009-exec-11] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun 
 java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)

atlassian-bitbucket.log に次のログが記録されます。

2015-06-29 05:17:36,139 ERROR [ajp-nio-127.0.0.1-8009-exec-31] xxxx @xxxx 127.0.0.1 "GET /scm/something/something.git/info/refs HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler something/something_shared[292]: Read request from 127.0.0.1 failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 255
The following was written to stderr:
error: cannot fork() for git-http-backend: Resource temporarily unavailable 

診断

環境

  • Large Enterprise server with similar environment as follows:

     <cpus>64</cpus>

診断ステップ

  • Enable git debug with below will show the debug logging stuck while trying to identity file .ssh cert

    export GIT_TRACE_PACKET=1
    export GIT_TRACE=1
    git clone ssh://admin@<mybitbucket.com>/something/something.git

原因

The default max thread on Bitbucket Server 3.10.x is too high for an Enterprise server with lots of hyper-threaded cores. This changes was made for BSERV-6868 - Getting issue details... STATUS to fix insufficient threads. However, this is then to be an issue when internal thread pool can grow up to 100 + 20 * 64 = 1380 threads which at some point the server could not create any more threads because it ran out of (virtual) memory.

The following are the default set per Bitbucket Server version:

3.11.0: executor.max.threads=${scaling.concurrency}
3.6.0 - 3.10.x: executor.max.threads=100+20*${scaling.concurrency} (where the later is # of cpu)
2.11.x: executor.max.threads=100

 

回避策

This is a known bug and a workaround would be to re-configure the default value of the executor.max.threads.

  • Shutdown Bitbucket Server
  • Add the following in bitbucket.properties

    executor.max.threads=2*cpu
    
  • Startup Bitbucket Server

ソリューション

This bug has been fixed in Bitbucket Server 3.11.0, see BSERV-7616 - Getting issue details... STATUS

Last modified on Mar 30, 2016

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.