Elasticsearch reports HTTP status 403 in bitbucket logs

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server と Data Center のみ - この記事は、サーバーおよびデータセンター プラットフォームのアトラシアン製品にのみ適用されます。

問題

With Elasticsearch version 6.2 or higher, code search may fail with the following error in Bitbucket log, atlassian-bitbucket.log:

com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 403. Response: IndexResponse{statusCode=403, content={
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
  },
  "status": 403
}}

診断

環境

  • Elasticsearch version 6.5.3 or higher

  • Bitbucket 6.0 or higher

原因

When Elasticsearch detects disk is running low on space, it puts itself into read-only mode. This decision is based on the percentage of free space on the disk, so on large volumes and disks, this error can manifest even if several gigabytes of disk space is free.

The default is 95% i.e. at least 5% of disk space must be free.

ソリューション

The bundled Elasticsearch does not accept requests from outside localhost. The below curl commands must be run from the same server that is hosting the Bitbucket application

Resolution #1

Free up disk space to ensure at least 5% of the disk is free. Once enough space is available, unlock the indices by issuing a command such as the following:

BITBUCKET_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

(warning) Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml


Resolution #2

Set the cluster.routing.allocation.disk.watermark.flood_stage value to a lower free percentage (or a fixed value). Here's an example of setting the value to a fixed amount of disk space:

Create a JSON file, say es.json, that contains the following:

{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "50gb",
    "cluster.routing.allocation.disk.watermark.high": "20gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "5gb"
  }
}

Then, run the following command to change threshold values:

BITBUCKET_HOST=localhost
ES_PORT=7992
JSON_FILE=es.json
curl -u bitbucketusername:bitbucketpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_cluster/settings -d@$JSON_FILE

Once done, run the following command to unlock the indices:

BITBUCKET_HOST=localhost
ES_PORT=7992
curl -u bitbucketusername:bitbucketpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
説明Free space threshold error in Elasticsearch
製品Bitbucket

最終更新日 2020 年 5 月 29 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.