要約

This page describes what to do when OpenSearch switches to read-only mode due to low disk space.

環境

Bitbucket 7.21.x, 8.x

原因

OpenSearch has a built-in safety feature that switches it to read-only mode if it thinks the disk is running out of space. This is based on the percentage of free space left on the disk. This means that, especially on big disks, you might get a warning even when there's still a lot of space (several gigabytes) left. By default, OpenSearch sets this warning to go off when 85% of the disk is used up, meaning it wants to keep at least 15% free at all times.

You might see the below entries in the logs: 

atlassian-bitbucket.log

"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "index [bitbucket-repository] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
}

bitbucket-search.log

[2024-03-01T14:23:08,108][WARN ][o.o.c.r.a.DiskThresholdMonitor] [bitbucket_bundled] high disk watermark [85%] exceeded on [JsyFQNNbRVK54GdbSpN_Og][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 22.3gb[9.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2024-03-01T14:24:08,109][WARN ][o.o.c.r.a.DiskThresholdMonitor] [bitbucket_bundled] high disk watermark [85%] exceeded on [JsyFQNNbRVK54GdbSpN_Og][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 22.3gb[9.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete 

ソリューション

ソリューション 1

Increase disk space on the search server or if it's a bundled search then increase Bitbucket node disk space.

ソリューション 2

Increase the disk usage watermark in percentage.

To do this run the below API calls:

• First GET the Cluster settings.

  curl  --user <Open search username>:<password> -H "Accept:application/json" -X GET "<opensearch-base-url>/_cluster/settings"

• Now run the below PUT call to set the watermark values

  curl  --user <OpenSearch_user name>:<password> -H "Content-Type: application/json" -X PUT -d '{
    "persistent": {
      "cluster.routing.allocation.disk.watermark.low": "85%",
      "cluster.routing.allocation.disk.watermark.high": "90%",
      "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
    }
  }' "<opensearch-base-url>/_cluster/settings"

• Now again run the GET call (first call) to confirm that the configured values are set.

ソリューション 3

Set the watermark in fixed GBs.

OpenSearch gives a warning based on the percentage of free space left, not the actual amount. So, you might get a warning even when there's lots of space left if your disk is big. To avoid this, you can set the warning to happen at a specific amount of space left, like a certain number of gigabytes (GB).

We have to use the same APIs as we did above.

• First GET the cluster settings.

  curl  --user <Open search username>:<password> -H "Accept:application/json" -X GET "<opensearch-base-url>/_cluster/settings"

• Now run the below PUT call to set the watermark values in fixed GB Values.

  curl  --user <OpenSearch_user name>:<password> -H "Content-Type: application/json" -X PUT -d '{
    "persistent": {
      "cluster.routing.allocation.disk.watermark.low": "25gb",
      "cluster.routing.allocation.disk.watermark.high": "15gb",
      "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"
    }
  }' "<opensearch-base-url>/_cluster/settings"

• Now again run the GET call(First call) to confirm that the configured values are set