Bitbucket repository corruption on remote mesh

Summary

This article addresses the issue of corrupted repositories on one or more remote mesh nodes in a Bitbucket 8.x Data Center environment.

Diagnosis

Confirm the node is corrupted

1. Access the remote mesh nodes and navigate to the path of the repository that is out of sync.

   1. To locate a mesh repository on a specific mesh node, check out How to find the Repository ID, Hierarchy ID, Partition ID and Repository location on disk for Mesh Repositories

2. Navigate to the repository that’s broken on disk:

   1 cd /var/atlassian/application-data/mesh/store/data/<partition>/<hierarchy>/repos/<id>

3. Execute:

   1 git fsck --no-dangling --connectivity-only

4. If the repository is corrupt, you should see:

   1 2 3 4 5 6 7 8 9 10 11 12 ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store/data/005e/22a016b54cc960cc427c/repos/13$ git fsck --no-dangling Checking object directories: 100% (256/256), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (193/193), done. broken link from commit f75870f707c80043c290a6af41c05f54faa6723e to commit b529d843c504552412b98eb7c2940ba6fc4703c4 missing commit b529d843c504552412b98eb7c2940ba6fc4703c4 Verifying commits in commit graph: 100% (1/1), done. Verifying commits in commit graph: 100% (1/1), done. Verifying commits in commit graph: 100% (21/21), done. Verifying commits in commit graph: 100% (1/1), done. Verifying commits in commit graph: 100% (21/21), done.

5. From the mesh logs on the same node, you can see the errors:

   1 2 3 4 5 6 7 8 9 10 From the mesh logs shared by the customer, we note the gc thread running and failing with below errors: 2025-04-10 00:16:45,276 WARN [git-gc:thread-3:p/0056/h/d43f19ae7cf916190781/r/shared-objects] - c.a.b.m.g.g.DefaultGarbageCollectionManager [d43f19ae7cf916190781] Abandoning garbage collection after 3 failed attempts com.atlassian.bitbucket.mesh.git.exception.CommandFailedException: [git fetch --prune --quiet --no-tags --negotiation-tip=refs/r/4514/refs/** ../4514 +refs/*:refs/r/4514/refs/*] exited with code 128 saying: error: Could not read 13e5bbc3c633db9ff1d478ba9a9f3482531d7783 remote: error: Could not read 13e5bbc3c633db9ff1d478ba9a9f3482531d7783 remote: fatal: revision walk setup failed error: git upload-pack: git-pack-objects died with error. fatal: git upload-pack: aborting due to possible repository corruption on the remote side. remote: aborting due to possible repository corruption on the remote side.^@ fatal: protocol error: bad pack header at com.atlassian.bitbucket.mesh.git.GitProcessCompletionHandler.onError(GitProcessCompletionHandler.java:228)

Cause

N/A.

Solution

Copy a healthy node to fix corrupt objects

Mesh won't automatically fix corrupt objects. You can restore them using a developer's copy of the repository data or from a healthy mesh node.

To find a healthy mesh node, check other nodes for the missing object using a loop with the missing hash:

For example, here you can find the missing object from node 2:

You can use the rsync method to resync git data from a healthy mesh node to fix corrupt repositories. Just ensure you schedule a maintenance window for Bitbucket and mesh systems.

If you find several corruptions and don't know how many repositories are affected, follow the steps below to fix the issue. With this approach, you can keep mesh nodes running smoothly without any downtime or maintenance breaks:

1. Select one node (node 1) that has corrupt repositories. In the example, one corrupt repository is present on node 2:

   1 2 3 4 5 6 7 8 9 10 ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store/data/005e/22a016b54cc960cc427c/repos/13/objects$ git fsck --no-dangling Checking object directories: 100% (256/256), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (157/157), done. broken link from commit b90ab2fec45b310f7dfb641cbed0465e7232e314 to tree 445bfdbd0d6ac01990e7a73692e98f011a7fd5f9 missing tree 445bfdbd0d6ac01990e7a73692e98f011a7fd5f9 Verifying commits in commit graph: 100% (3/3), done. Verifying commits in commit graph: 100% (40/40), done. Verifying commits in commit graph: 100% (43/43), done.

2. Create a backup of the store/data folder on the node with corrupt repositories, then rename the current data folder to data_bak.

   1 2 3 4 5 6 7 8 9 ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store$ sudo mv data data.back ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store$ ls -ltr total 20 drwxr-x--- 2 bitbucket bitbucket 4096 Apr 11 11:38 work-trees drwxr-x--- 3 bitbucket bitbucket 4096 Apr 11 11:38 hook-scripts drwxr-x--- 4 bitbucket bitbucket 4096 Apr 11 11:38 config -rw-r----- 1 bitbucket bitbucket 86 Apr 11 11:38 store.properties drwxr-x--- 3 bitbucket bitbucket 4096 Apr 14 13:35 data.back ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store$

3. This will essentially cover all inconsistent repositories with missing git objects. Before proceeding, make sure to back up the data directory.

   1. Only do this for the store/data directory, as the other directories contain important files needed for communication with Bitbucket, like the mesh signing keys

4. Now, run the Verify Consistency API. In this example, since there are two inconsistent repositories, the output looks like this:

   1 2 3 4 5 curl -k -X POST --location "https://linux-97773.prod.atl-cd.net/bitbucket/rest/ui/latest/admin/git/mesh/troubleshooting/verify-consistency" \ -H "Content-Type: application/json" \ -H "Accept: application/json" \ --basic --user "admin:sphere" [{"repository":"~ADMIN/test-repo[1]","inconsistencies":["The replica of ~ADMIN/test-repo on Bitbucket Mesh node 2 is inconsistent and has been scheduled for repair"]},{"repository":"GOL/game[13]","inconsistencies":["The replica of GOL/game on Bitbucket Mesh node 2 is inconsistent and has been scheduled for repair"]}]

   1. The time it takes depends on how many and how large the inconsistent repositories are on the node with the renamed data folder

5. Once complete, run the git fsck command again in any the corrupt repository path to confirm it's fixed:

   1 git fsck --no-dangling --connectivity-only

   Example:

   1 2 3 4 5 6 7 ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store/data/005e/22a016b54cc960cc427c/repos/13$ git fsck --no-dangling Checking object directories: 100% (256/256), done. Checking object directories: 100% (256/256), done. Checking objects: 100% (156/156), done. Verifying commits in commit graph: 100% (45/45), done. Verifying commits in commit graph: 100% (45/45), done. ubuntu@ip-10-140-18-197:~/bitbucket-mesh2/store/data/005e/22a016b54cc960cc427c/repos/13$