Confluence Unresponsive due to Blocked Threads when Comala Workflows Plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 Has Been Installed
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Confluence becomes gradually unresponsive under load when Comala Workflows plugin 5.5.0, 5.5.1, 5.5.2, 5.7.2 or 5.7.3 has been installed on this instance.
Diagnosis
Troubleshooting Confluence performance issues with thread dumps taken during the time of incident reveal that most HTTP threads (look for those with "-exec-" in its thread name) are in BLOCKED status, all "waiting to lock" the same object and each containing a reference to org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton, for example:
1
2
3
4
5
6
7
8
9
"http-nio-8093-exec-302" #12461 daemon prio=5 os_prio=0 tid=0x00007f4058160000 nid=0x215d waiting for monitor entry [0x00007f3f7e8dc000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
- waiting to lock <0x0000000462c907c8> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
...
Tomcat logs at <confluence_install>/logs/catalina.out may also report many stuck threads with similar stacktraces, for instances:
1
2
3
4
5
6
7
8
9
10
18-Jul-2018 20:45:03.998 WARNING [ContainerBackgroundProcessor[StandardEngine[Standalone]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8093-exec-149" (id=12303) has been active for 68,146 milliseconds (since 7/18/18 8:43 PM) to serve the same request for http://confluence.example.com/display/SPACEKEY/Example+Page and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 68 thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:489)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:447)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:423)
at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:220)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1177)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1116)
...Cause
Comala Workflows plugin versions 5.5.0, 5.5.1, 5.5.2, 5.7.2 and 5.7.3 contained a bug which can cause performance/stability issues given sufficient load. This issue has been resolved in plugin version 5.5.3 and 5.8.1.
Solution
Resolution
Upgrade to plugin version 5.5.3 or 5.8.1 and above. If you are unable to upgrade right away, disabling the existing Workflows plugin may be used as a temporary workaround.
Was this helpful?