Jira サーバーの Too many open files エラー

関連コンテンツ

  • 関連コンテンツがありません

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

症状

A Jira application experiences a general loss of functionality in several areas.

atlassian-jira.log ログに次のエラーが表示される。

java.io.IOException: java.io.IOException: Too many open files

現在のオープン ファイル ハンドラの上限を確認するには、次のコマンドを実行します。

ulimit -aS | grep open

その後、次のような行を確認します。

open files                      (-n) 2560

診断

オプション 1

In order to identify the open files that are completely unlinked, the lsof +L1 command can be used, for example:

lsof +L1 > open_files.txt
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NLINK     NODE NAME
java     2565  dos  534r   REG   8,17    11219     0 57809485 /home/dos/deploy/applinks-jira/temp/jar_cache3983695525155383469.tmp (deleted)
java     2565  dos  536r   REG   8,17    29732     0 57809486 /home/dos/deploy/applinks-jira/temp/jar_cache5041452221772032513.tmp (deleted)
java     2565  dos  537r   REG   8,17   197860     0 57809487 /home/dos/deploy/applinks-jira/temp/jar_cache6047396568660382237.tmp (deleted)


オプション 2

Linux kernel exports information about all of the running processes on a Linux system via the proc pseudo file system, usually mounted at /proc.
In the /proc pseudo file system, we can find the open file descriptors under /proc/<pid>/fd/ where <pid> is the JAVA PID.

To count the open files opened by the JAVA process, we can use the command below to get a count:

ls -U /proc/\{JAVA-PID}/fd | wc -l

 or to show files:

ls -lU /proc/\{JAVA-PID}/fd | tail -n+2


上記のソリューションが動作しない場合はアトラシアン サポートで詳細な調査を行うことができます。調整を依頼するには次のものをアトラシアン サポートに提供してください。

  1. The output of Option 1 and/or Option 2
    (warning) The commands must be executed by the JIRA user, or a user who can view the files. For example, if Jira is running as root (which is not at all recommended), executing this command as jira will not show the open files.
  2. 例外が返されたタイミングで取得されたヒープ ダンプ。詳細については「ヒープ ダンプの生成」をご確認ください。
  3. A Jira application Support ZIP.

原因

UNIX systems have a limit on the number of files that can be concurrently open by any one process. The default for most distributions is only 1024 files, and for certain configurations of Jira applications, this is too small a number. When that limit is hit, the above exception is generated and Jira applications can fail to function as it can't open the required files to complete its current operation.

There are certain bugs in the application that are known to cause this behavior:

  • JRA-29587 - 課題詳細を取得中... ステータス
  • JRA-35726 - 課題詳細を取得中... ステータス
  • JRA-39114 - Getting issue details... STATUS (affects Jira 6.2 and higher)
  • JRASERVER-68653 - 課題詳細を取得中... ステータス

We have the following improvement requests to better handle this in Jira applications:

  • JRA-33887 - 課題詳細を取得中... ステータス
  • JRA-33463 - 課題詳細を取得中... ステータス

ソリューション


These changes will only work on an installation that uses built-in initd script for starting Jira. For installations that use custom build service for systemd (latest versions of linux OS-es) changes will need to be applied directly in that systemd service configuration (ie, update the /usr/lib/systemd/system/jira.service file, followed by systemctl daemon-reload) in a form of:

[Service]
LimitNOFILE=20000

In most cases, setting the ulimit in Jira's setenv.sh should work:

  1. If the $JIRA_HOME/caches/indexes folder is mounted over NFS move it to a local mount (i.e. storage on the same server as the Jira instance). NFS is not supported as per our Jira application Supported Platforms and will cause this problem to occur at a much higher frequency.
  2. Jira アプリケーションを停止します。

  3. $JIRA_INSTALL/bin/setenv.sh を編集し、次の内容をファイル上部に含めます。

    ulimit -n 16384

    This will set that value each time Jira applications are started, however, it will need to be manually migrated when upgrading JIRA applications.

  4. Start your Jira application.
  5. /proc/<pid>/limits  を実行することで変更を確認できます。ここで <pid> はアプリケーションのプロセス ID です。

If you are using a Jira application version with the bug in JRA-29587 - Getting issue details... STATUS , upgrade it to the latest version. If using NFS, migrate to a local storage mount.

一部のオペレーティング システムでは、制限の設定用に追加の構成が必要な場合があります。

For most Linux systems you would modify the limits.conf file:

  1. To modify the limits.conf file, use the following command:
    sudo vim /etc/security/limits.conf 

  2. Add/Edit the following for the user that runs Jira application. If you have used the bundled installer, this will be jira.

    limits.conf 
    #<domain>      <type>  <item>         <value>
#
#*               soft    core            0
#root            hard    core            100000
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4
jira           soft    nofile          16384
jira           hard    nofile          32768 

  3. common-session ファイルを次のように変更します。

    sudo vim /etc/pam.d/common-session
    (info) common-session is a file only available in debian/ubuntu.

  4. 次の行を追加します。

    common-session 
    # The following changes were made from the JIRA KB (https://confluence.atlassian.com/display/JIRAKB/Loss+of+Functionality+due+to+Too+Many+Open+Files+Error):
session required pam_limits.so

In some circumstances it is necessary to check and configure the changes globally, this is done in 2 different places, it is advised to check all of this for what is in effect on your system and configure accordingly:

  1. The first place to check is the systctl.conf file.
    1. Modify the systctl.conf file with the following command:
      sudo vim /etc/sysctl.conf

    2. Add the following to the bottom of the file or modify the existing value if present:

      sysctl.conf 
      fs.file-max=16384

    3. Users will need to logout and login again for the changes to take effect. If you want to apply the limit immediately, you can use the following command:


    4. sysctl -p
  2. The other place to configure this is within the sysctl.d directory, the filename can be anything you as long as it follows with ".conf" e.g. "30-jira.conf", the number here helps to give an order of precedence see the previous link for more info.
    1. To create/edit the file use the following:
    2. sudo vim /etc/sysctl.d/30-jira.conf

    3. Add the following to the bottom of the file or modify the existing value if present:

      30-jira.conf 
      fs.file-max=16384

  3. (info) N.B., for RHEL/CentOS/Fedora/Scientific Linux, you'll need to modify the login file with the following:

    sudo vim /etc/pam.d/login

  4. Add the following line to the bottom of the file:

    ログイン 
    # The following changes were made from the JIRA KB (https://confluence.atlassian.com/display/JIRAKB/Loss+of+Functionality+due+to+Too+Many+Open+Files+Error):
session required pam_limits.so
  5. See this external blog post for a more detailed write-up on configuring the open file limits in Linux: Linux Increase The Maximum Number Of Open Files / File Descriptors (FD)

N.B. For any other operating system (please see your operating system's manual for details).

Lastly, restart the Jira server application to take effect.

tip/resting Created with Sketch.

For exceedingly large instances, we recommend consulting with our partners for scaling Jira applications. 

最終更新日: 2024 年 1 月 12 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する

関連コンテンツ

  • 関連コンテンツがありません
Powered by Confluence and Scroll Viewport.