Jira is unresponsive either consistently or randomly due to stuck threads and / or thread pool exhaustion

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

プラットフォームについて: Data Center のみ - この記事は、Data Center プラットフォームのアトラシアン製品にのみ適用されます。

この KB は Data Center バージョンの製品用に作成されています。Data Center 固有ではない機能の Data Center KB は、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。

*Fisheye および Crucible は除く

要約

At times Jira performance may slowly deteriorate and appear as unresponsive.

環境

Any Jira version on Windows or Linux

診断

  • Tomcat log file catalina.out in $JIRA_INST/logs/ will show many exceptions like below:

    26-Oct-2022 17:41:42.110 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8080-exec-75] (id=[438]) has been active for [121,849] milliseconds (since [10/26/22 5:39 PM]) to serve the same request for https://URL/plugins/servlet/nfj/PushNotification and may be stuck (configured threshold for this StuckThreadDetectionValve is [120] seconds). There is/are [45] thread(s) in total that are monitored by this Valve and may be stuck.
        java.lang.Throwable
            at com.infosysta.jira.nfj.servlet.Servlet_PushNotification.doGet(Servlet_PushNotification.java:30)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

    (info) Notice in the above example /servlet/nfj/PushNotification is just one example, it can be anything, a good giveaway here is that there's a high number of of this threads (45) that maybe stuck

  • Generate at least 6 thread dumps, waiting 10 seconds in between: Troubleshooting Jira performance with Thread dumps; Once generated, inspect the dumps to try and identify long running threads amongst those dumps (Feel free to log a support request with Atlassian Support to assist with that)
  • Inspect one of the thread dumps and count how many http threads are there (simple search for 'http-' in a text editor to count the number of matches will do). Compare that number to the maxThreads value defined in the main Tomcat configuration file - server.xml; If the number of http threads in a thread dump is higher than the maximum number of threads configured in Tomcat - that means that the maximum thread pool has maxed out

原因

A third party application is holding onto all of Tomcat's threads, in the above log example it was due to "In-app & Desktop Notifications for Jira" add-on, however, it could be other add-ons;

These threads could also eventually cause Tomcat running out of available threads, whereby the number of http threads in the application will exceed the maximum number of threads that the Tomcat is configured to handle. The application will still allow new threads to queue up, however, they will have to wait for the other threads to complete, thus slowing down the application significantly.

The above is only a high level cause, to identify the culprit, further, in depth investigation maybe required


ソリューション

  • Once the suspect add-on has been identified  - try and disable that add-on (if possible) to see if the performance has improved. If it has - please reach out to the add-on vendor for further guidance;
  • If thread pool exhaustion is identified, potentially maxThreads= value in server.xml maybe increased slightly (depending on the number of threads you see in the thread dumps) but that will need to be done with caution as the higher this number is, the more resources Jira will need to process the number of threads comfortably without causing performance bottlenecks in other parts of the application, i.e. along with increasing maxThreads you may need to increase the heap size (xmx value) and DB pool size (dbconfig.xml). Reach out to Atlassian Support if in doubt.



最終更新日 2022 年 11 月 10 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.