Bitbucket Server stops responding due to a Java-level deadlock

関連コンテンツ

  • 関連コンテンツがありません

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

症状

Bitbucket Server becomes unresponsive.

The sequence of events that lead to this situation are not yet known and it is not easily reproducible.

環境

The issue affects instances using Java 8.

スレッド ダンプ

Thread dumps taken during this time would show Java-level deadlocks.

The deadlock is encountered between a ClientPoller, BlockPoller and a few other http/s threads on the following methods:

  • java.nio.channels.spi.AbstractSelectableChannel.removeKey
  • java.nio.channels.spi.AbstractSelector.cancel

Click here for a sample thread dump 
Found one Java-level deadlock:
=============================
"https-jsse-nio-8201-exec-222":
  waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-ClientPoller"
"https-jsse-nio-8201-ClientPoller":
  waiting to lock monitor 0x00007f8bb41791b8 (object 0x00000006c4342f48, a java.lang.Object),
  which is held by "https-jsse-nio-8201-exec-52"
"https-jsse-nio-8201-exec-52":
  waiting to lock monitor 0x00007f8c38306148 (object 0x00000006c5f6dcb0, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-BlockPoller"
"https-jsse-nio-8201-BlockPoller":
  waiting to lock monitor 0x00007f8b441b6568 (object 0x00000006c4343de0, a java.lang.Object),
  which is held by "https-jsse-nio-8201-exec-15"
"https-jsse-nio-8201-exec-15":
  waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-ClientPoller"

Java stack information for the threads listed above:
===================================================
"https-jsse-nio-8201-exec-222":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c43842d0> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c43842c0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c43842f8> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c43843a8> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4384308> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-ClientPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c6278be8> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c31a3fc0> (a sun.nio.ch.Util$3)
- locked <0x00000006c31a3fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c31a3f68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:706)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-exec-52":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c5f6dcb0> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4342f58> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4342f80> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4343030> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4342f90> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-BlockPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c5f6dcb0> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c3da4290> (a sun.nio.ch.Util$3)
- locked <0x00000006c3da4280> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c3da4238> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"https-jsse-nio-8201-exec-15":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4394e78> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4394ea0> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4394f50> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4394eb0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

原因

The deadlock is caused by the following bug in Tomcat versions 9.0.20 to 9.0.31:

64007 - A Deadlock with Poller run and cancelledKey

影響を受けるバージョン

The Bitbucket Server versions affected by this bug are:

Bitbucket バージョンEmbedded Tomcat Version
6.5.x9.0.21
6.6.x9.0.21
6.7.x9.0.21
6.8.x9.0.21
6.9.x9.0.21
6.10.0 - 6.10.29.0.27
7.0.0 - 7.0.19.0.30

回避策

Perform a restart of Bitbucket Server.
For multi-node Bitbucket Data Center instances, only the affected node needs to be restarted.

修正バージョン

This issue is fixed in Bitbucket Server versions 6.10.3+, 7.0.2+, 7.1.0+ or later.

These versions use Tomcat version 9.0.31+ or later.

Java 11

If an upgrade to a fix version of Bitbucket is not possible, an alternative is to use Java 11 where the issue with deadlocks is addressed (reference: Tomcat repository commit). Java 11 is installed by default with the Bitbucket 8.6.0 installer.


説明

Bitbucket Server stops responding due to a Java-level deadlock.

The issue is due to a bug in certain embedded Tomcat versions

製品Bitbucket Server

最終更新日 2022 年 11 月 23 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する

関連コンテンツ

  • 関連コンテンツがありません
Powered by Confluence and Scroll Viewport.