JIRA threads grow uncontrollably with lots of Analytics threads in Timed_Waiting state
プラットフォームについて: サーバーと Data Center のみ。この記事は、サーバーおよび Data Center プラットフォームのアトラシアン製品にのみ適用されます。
問題
JIRA threads grow uncontrollably (a few thousands) exceeding the maxThreads specified in the Connector used by JIRA in server.xml. Due to this, JIRA may run out of resources and throw this error:
21-Feb-2017 13:51:46.985 SEVERE [ajp-nio-8009-exec-148] org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at com.sun.jndi.ldap.Connection.<init>(Connection.java:234)
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137)診断
A lot of these threads are found in thread dumps: Analytics in Timed_Waiting state
"Analytics" prio=1 tid=0x0000000000004645 nid=0 sleeping
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at com.segment.backo.Backo.sleep(Backo.java:72)
at com.segment.analytics.internal.AnalyticsClient$BatchUploadTask.run(AnalyticsClient.java:158)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at com.segment.analytics.Platform$1$1.run(Platform.java:44)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- java.util.concurrent.ThreadPoolExecutor$Worker@@8a291d
原因
The thread above comes from Tempo Analytics, which is bundled in one of the Tempo plugins. The keyword that helps identify this, in this case, is com.segment.
The way to find which plugin is causing this is to unpack the installed plugins' .jar files and then grep for the class name in all the unpacked folders. E.g. for this case:
grep -r "com.segment" ./Be aware that ScriptRunner plugin also uses *com.segment.analytics* classes for analytics purposes.
回避策
Go to JIRA Administration -> Add-ons -> Tempo Analytics -> disable Analytics. This will prevent the Analytics threads from accumulating.
ソリューション
According to Tempo for JIRA, the issue is fixed in Tempo Timesheets 8.1.3. Release notes can be found here.