Bitbucket Server stops responding due to a Java-level deadlock

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server と Data Center のみ - この記事は、サーバーおよびデータセンター プラットフォームのアトラシアン製品にのみ適用されます。

症状

Bitbucket Server becomes unresponsive.

The sequence of events that lead to this situation are not yet known and it is not easily reproducible.

環境

The issue affects instances using Java 8.

スレッド ダンプ

Thread dumps taken during this time would show Java-level deadlocks.

The deadlock is encountered between a ClientPollerBlockPoller and a few other http/s threads on the following methods:

  • java.nio.channels.spi.AbstractSelectableChannel.removeKey
  • java.nio.channels.spi.AbstractSelector.cancel

Click here for a sample thread dump
Found one Java-level deadlock:
=============================
"https-jsse-nio-8201-exec-222":
  waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-ClientPoller"
"https-jsse-nio-8201-ClientPoller":
  waiting to lock monitor 0x00007f8bb41791b8 (object 0x00000006c4342f48, a java.lang.Object),
  which is held by "https-jsse-nio-8201-exec-52"
"https-jsse-nio-8201-exec-52":
  waiting to lock monitor 0x00007f8c38306148 (object 0x00000006c5f6dcb0, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-BlockPoller"
"https-jsse-nio-8201-BlockPoller":
  waiting to lock monitor 0x00007f8b441b6568 (object 0x00000006c4343de0, a java.lang.Object),
  which is held by "https-jsse-nio-8201-exec-15"
"https-jsse-nio-8201-exec-15":
  waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
  which is held by "https-jsse-nio-8201-ClientPoller"

Java stack information for the threads listed above:
===================================================
"https-jsse-nio-8201-exec-222":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c43842d0> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c43842c0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c43842f8> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c43843a8> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4384308> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-ClientPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c6278be8> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c31a3fc0> (a sun.nio.ch.Util$3)
- locked <0x00000006c31a3fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c31a3f68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:706)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-exec-52":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c5f6dcb0> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4342f58> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4342f80> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4343030> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4342f90> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-BlockPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c5f6dcb0> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c3da4290> (a sun.nio.ch.Util$3)
- locked <0x00000006c3da4280> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c3da4238> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"https-jsse-nio-8201-exec-15":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4394e78> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4394ea0> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4394f50> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4394eb0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

原因

The deadlock is caused by the following bug in Tomcat versions 9.0.20 to 9.0.31:

64007 - A Deadlock with Poller run and cancelledKey

影響バージョン

The Bitbucket Server versions affected by this bug are:

Bitbucket バージョンEmbedded Tomcat Version
6.5.x9.0.21
6.6.x9.0.21
6.7.x9.0.21
6.8.x9.0.21
6.9.x9.0.21
6.10.0 〜 6.10.29.0.27
7.0.0 〜 7.0.19.0.30

回避策

Perform a restart of Bitbucket Server.
For multi-node Bitbucket Data Center instances, only the affected node needs to be restarted.

修正バージョン

This issue is fixed in Bitbucket Server versions 6.10.3+, 7.0.2+, 7.1.0+ or later.

These versions use Tomcat version 9.0.31+ or later.

Java 11

If an upgrade to a fix version of Bitbucket is not possible, an alternative is to use Java 11 where the issue with deadlocks is addressed. (reference: Tomcat repository commit)


説明

Bitbucket Server stops responding due to a Java-level deadlock.

The issue is due to a bug in certain embedded Tomcat versions

製品Bitbucket Server

最終更新日 2020 年 5 月 27 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.