Confluence Unresponsive due to Blocked Threads when Comala Workflows Plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 Has Been Installed
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
問題
Confluence becomes gradually unresponsive under load when Comala Workflows plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 has been installed on this instance.
診断
Thread dumps taken during the time of incident reveal that most HTTP threads (look for those with "-exec-" in its thread name) are in BLOCKED status, all "waiting to lock" the same object and each containing a reference to org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton, for example:
"http-nio-8093-exec-302" #12461 daemon prio=5 os_prio=0 tid=0x00007f4058160000 nid=0x215d waiting for monitor entry [0x00007f3f7e8dc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x0000000462c907c8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
...
Tomcat logs at <confluence_install>/logs/catalina.out may also report many stuck threads with similar stacktraces, for instances:
18-Jul-2018 20:45:03.998 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8093-exec-149" (id=12303) has been active for 68,146 milliseconds (since 7/18/18 8:43 PM) to serve the same request for http://confluence.example.com/display/SPACEKEY/Example+Page and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 68 thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1177)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
...
原因
Comala Workflows plugin versions 5.5.0, 5.5.1, 5.5.2, 5.7.2 and 5.7.3 contained a bug which can cause performance/stability issues given sufficient load. This issue has been resolved in plugin version 5.5.3 and 5.8.1.
ソリューション
Upgrade to plugin version 5.5.3 or 5.8.1 and above. If you are unable to upgrade right away, disabling the existing Workflows plugin may be used as a temporary workaround.