Elasticsearch reports HTTP status 403 in bitbucket logs
プラットフォームについて: Server と Data Center のみ - この記事は、サーバーおよびデータセンター プラットフォームのアトラシアン製品にのみ適用されます。
問題
With Elasticsearch version 6.2 or higher, code search may fail with the following error in Bitbucket log, atlassian-bitbucket.log:
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 403. Response: IndexResponse{statusCode=403, content={
"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
}
],
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
},
"status": 403
}}
診断
環境
Elasticsearch version 6.5.3 or higher
- Bitbucket 6.0 or higher
原因
When Elasticsearch detects disk is running low on space, it puts itself into read-only mode. This decision is based on the percentage of free space on the disk, so on large volumes and disks, this error can manifest even if several gigabytes of disk space is free.
The default is 95% i.e. at least 5% of disk space must be free.
ソリューション
The bundled Elasticsearch does not accept requests from outside localhost. The below curl commands must be run from the same server that is hosting the Bitbucket application
Resolution #1
Free up disk space to ensure at least 5% of the disk is free. Once enough space is available, unlock the indices by issuing a command such as the following:
BITBUCKET_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml
Resolution #2
Set the cluster.routing.allocation.disk.watermark.flood_stage value to a lower free percentage (or a fixed value). Here's an example of setting the value to a fixed amount of disk space:
Create a JSON file, say es.json, that contains the following:
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "50gb",
"cluster.routing.allocation.disk.watermark.high": "20gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "5gb"
}
}Then, run the following command to change threshold values:
BITBUCKET_HOST=localhost
ES_PORT=7992
JSON_FILE=es.json
curl -u bitbucketusername:bitbucketpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_cluster/settings -d@$JSON_FILEOnce done, run the following command to unlock the indices:
BITBUCKET_HOST=localhost
ES_PORT=7992
curl -u bitbucketusername:bitbucketpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'