After upgrading Bitbucket Server, bundled Elasticsearch has been crashing while indexing repositories

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問


プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

要約

After upgrading Bitbucket Server, the bundled Elasticsearch has been systematically crashing when attempting to index repositories

環境

This issue may affect multiple versions of Bitbucket Server.
For example, it has been seen after the upgrade from version 6.10.0 to 7.6.0.

診断

In atlassian-bitbucket.log you may find errors similar to the following:

2021-03-01 01:23:45,678 ERROR [search-indexing:thread-1]  c.a.b.i.s.i.e.DefaultIndexEventWorker Indexing - Failed for project PROJO1 (id: 1001) with error: Index response returned status code 429. Response: IndexResponse{statusCode=429, content={
  "error": {
    "root_cause": [
      {
        "type": "circuit_breaking_exception",
        "reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
        "bytes_wanted": 1034946678,
        "bytes_limit": 986061209,
        "durability": "PERMANENT"
      }
    ],
    "type": "circuit_breaking_exception",
    "reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
    "bytes_wanted": 1034946678,
    "bytes_limit": 986061209,
    "durability": "PERMANENT"
  },
  "status": 429
}}
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 429. Response: IndexResponse{statusCode=429, content={
  "error": {
    "root_cause": [
      {
        "type": "circuit_breaking_exception",
        "reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
        "bytes_wanted": 1034946678,
        "bytes_limit": 986061209,
        "durability": "PERMANENT"
      }
    ],
    "type": "circuit_breaking_exception",
    "reason": "[parent] Data too large, data for [<http_request>] would be [1034946678/987mb], which is larger than the limit of [986061209/940.3mb], real usage: [1034946568/987mb], new bytes reserved: [110/110b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=110/110b, accounting=41161613/39.2mb]",
    "bytes_wanted": 1034946678,
    "bytes_limit": 986061209,
    "durability": "PERMANENT"
  },
  "status": 429
}}


原因

Bundled Elasticsearch is running with insufficient JVM heap.
If Elasticsearch was previously configured to allow a larger heap, that custom configuration has been overwritten by the new Bitbucket version.

ソリューション

  1. Verify what was the maximum heap size defined for Elasticsearch before the upgrade:
    1. Navigate to $BITBUCKET_HOME/search/config
    2. Open a pre-upgrade copy of the jvm.options file.
      If you don't have it, you may need to restore it from pre-upgrade backup into a location outside of $BITBUCKET_HOME and open it there.

  2. Find the previously used heap settings defined in the 2 separate lines.
    One of them should start with -Xms and the other with -Xmx - eg.

    -Xms4g
    -Xmx4g
  3. Edit the current $BITBUCKET_HOME/search/config/jvm.options file and find the equivalent configuration lines, with the default values of 1g:

    -Xms1g
    -Xmx1g

    Replace both lines with those you found in the pre-upgrade copy of this file.

  4. Save the modified jvm.options file

  5. Restart your Bitbucket Server, so the bundled Elasticsearch also gets restarted, and the new configuration is applied.

  6. Confirm that the errors mentioned above do not occur any longer.



最終更新日 2021 年 5 月 3 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.