How to set disk usage watermark in OpenSearch for Bitbucket Server and Datacenter
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
This page describes what to do when OpenSearch switches to read-only mode due to low disk space.
Environment
Bitbucket 7.21.x, 8.x
Cause
OpenSearch has a built-in safety feature that switches it to read-only mode if it thinks the disk is running out of space. This is based on the percentage of free space left on the disk. This means that, especially on big disks, you might get a warning even when there's still a lot of space (several gigabytes) left. By default, OpenSearch sets this warning to go off when 85% of the disk is used up, meaning it wants to keep at least 15% free at all times.
You might see the below entries in the logs:
atlassian-bitbucket.log
1
2
3
4
5
6
"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "index [bitbucket-repository] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"
}bitbucket-search.log
1
2
[2024-03-01T14:23:08,108][WARN ][o.o.c.r.a.DiskThresholdMonitor] [bitbucket_bundled] high disk watermark [85%] exceeded on [JsyFQNNbRVK54GdbSpN_Og][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 22.3gb[9.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2024-03-01T14:24:08,109][WARN ][o.o.c.r.a.DiskThresholdMonitor] [bitbucket_bundled] high disk watermark [85%] exceeded on [JsyFQNNbRVK54GdbSpN_Og][bitbucket_bundled][/data/bitbucket/shared/search/data/nodes/0] free: 22.3gb[9.9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete Solution
Solution 1
Increase disk space on the search server or if it's a bundled search then increase Bitbucket node disk space.
Solution 2
Increase the disk usage watermark in percentage.
To do this run the below API calls:
First GET the Cluster settings.
1curl --user <Open search username>:<password> -H "Accept:application/json" -X GET "<opensearch-base-url>/_cluster/settings"Now run the below PUT call to set the watermark values
1 2 3 4 5 6 7curl --user <OpenSearch_user name>:<password> -H "Content-Type: application/json" -X PUT -d '{ "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%", "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } }' "<opensearch-base-url>/_cluster/settings"Now again run the GET call (first call) to confirm that the configured values are set.
Solution 3
Set the watermark in fixed GBs.
OpenSearch gives a warning based on the percentage of free space left, not the actual amount. So, you might get a warning even when there's lots of space left if your disk is big. To avoid this, you can set the warning to happen at a specific amount of space left, like a certain number of gigabytes (GB).
We have to use the same APIs as we did above.
First GET the cluster settings.
1curl --user <Open search username>:<password> -H "Accept:application/json" -X GET "<opensearch-base-url>/_cluster/settings"Now run the below PUT call to set the watermark values in fixed GB Values.
1 2 3 4 5 6 7curl --user <OpenSearch_user name>:<password> -H "Content-Type: application/json" -X PUT -d '{ "persistent": { "cluster.routing.allocation.disk.watermark.low": "25gb", "cluster.routing.allocation.disk.watermark.high": "15gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb" } }' "<opensearch-base-url>/_cluster/settings"Now again run the GET call(First call) to confirm that the configured values are set
Note: You can find the OpenSearch user name and base URL by navigating to Bitbucket Administration -> Server settings -> Search -> Search server username, Search server URL. If you are unaware of the password then you would have to follow this link and reset the password, after resetting the password run the above curl commands.
Was this helpful?