Elasticsearch reports HTTP status 403 in bitbucket logs
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
要約
With Elasticsearch version 6 or higher, code search may stop indexing with index failures.
環境
Elasticsearch version 6 or higher
- Bitbucket 6.0 or higher
診断
The following error will be in atlassian-bitbucket.log during an index write operation:
com.atlassian.bitbucket.internal.search.indexing.exceptions.IndexException: Index response returned status code 403. Response: IndexResponse{statusCode=403, content={
"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
}
],
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
},
"status": 403
}}And one of the following may be visible in the Elasticsearch logs:
[o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]high disk watermark [85%] exceeded on [MqF6riY3TcyaM46vc7KWfg][bitbucket_bundled][C:\Atlassian\ApplicationData\Bitbucket\shared\search\data\nodes\0] free: 5.7gb[5.1%], shards will be relocated away from this node2020-02-04T00:00:22,306][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [1f1m2yepQ5Sti64cWwD6ZQ][bitbucket_bundled][/var/atlassian/application-data/bitbucket/shared/search/data/nodes/0] free: 17.6gb[11.8%], replicas will not be assigned to this node
[2020-02-04T00:00:52,348][INFO ][o.e.c.r.a.DiskThresholdMonitor] [bitbucket_bundled]low disk watermark [85%] exceeded on [1f1mq3e5Q5Sti6pcWwD6ZQ][bitbucket_bundled][/var/atlassian/application-data/bitbucket/shared/search/data/nodes/0] free: 17.5gb[11.7%], replicas will not be assigned to this node原因
When Elasticsearch detects disk is running low on space, it puts itself into read-only mode. This decision is based on the percentage of free space on the disk, so on large volumes and disks, this error can manifest even if several gigabytes of disk space is free.
Elasticsearch sets a default of 85% of disk usage (i.e. at least 15% of disk space must be free)
Therefore, even though you may notice that your current free disk space is above the required free disk space, the indices could have been locked if it was running a search on a low disk space and a solution needs to be applied to unlock the indices.
ソリューション
The bundled Elasticsearch does not accept requests from outside localhost. The below curl commands must be run from the same server that is hosting the Bitbucket application
Solution #1
Free up disk space to ensure at least 15% of the disk is free. Once enough space is available, unlock the indices by issuing a command such as the following:
ES_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$ES_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml
Server instances by default have the bundled Elasticsearch. For remote Elasticsearch instances, please update ES_HOST and ES_PORT as appropriate.
Solution #2
Set the cluster.routing.allocation.disk.watermark.flood_stage value to a lower free percentage (or a fixed value). Here's an example of setting the value to a fixed amount of disk space:
Create a JSON file, say es.json, that contains the following:
{
"transient": {
"cluster.routing.allocation.disk.watermark.low": "50gb",
"cluster.routing.allocation.disk.watermark.high": "20gb",
"cluster.routing.allocation.disk.watermark.flood_stage": "5gb"
}
}Then, run the following command to change threshold values:
ED_HOST=localhost
ES_PORT=7992
JSON_FILE=es.json
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_cluster/settings -d@$JSON_FILEOnce done, run the following command to unlock the indices:
ES_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'In case you are using Windows to run the command syntax is the following:
ES_HOST=localhost
ES_PORT=7992
curl -u elasticsearchusername:elasticsearchpassword -XPUT -H "Content-Type: application/json" http://$BITBUCKET_HOST:$ES_PORT/_all/_settings -d "{\"index.blocks.read_only_allow_delete\": null}" Be sure to replace the elasticsearchusername and elasticsearchpassword fields above with the actual credentials you have configured for Elasticsearch in $BITBUCKET_HOME/shared/search/buckler/buckler.yml
Server instances by default have the bundled Elasticsearch. For remote Elasticsearch instances, please update ES_HOST and ES_PORT as appropriate.