High load on Hazelcast and Bitbucket due to failing GC on refs/pull-requests/<pr id>/v* files
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
High CPU and performance issues in Bitbucket appear along with the following messages in the atlassian-bitbucket-yyyy-mm-dd.log:
1
c.a.s.i.c.HazelcastBucketedExecutor Attempt 2 of 5 at processing bucket 'git:gc@385d519558f5df598f79' for executor 'git:gc' failed: ''/srv/git/current/bin/git gc' exited with code 255 saying: fatal: bad object refs/pull-requests/1/v3You may also see similar errors for the following fiile names:
merge-basemerge-conflictedmerge-cleannotesto
These could be accompanied by hundreds of Hazelcast errors in the logs as well.
Cause
Prior to Stash 2.9 the refs/pull-requests/<pr id>/v* files were used, and from 2.9 onwards we use them from the stash-refs directory instead. It's possible the upgrade tasks failed to remove the v* files or these files were restored from a backup. Garbage collection continuously failing garbage collection can put a server under a huge amount of load.
ℹ️ Bitbucket 5+ moves GC off the BucketedExecutor and Hazelcast. Garbage collection will only create load on HazelCast in Bitbucket 4.x versions.
Resolution
Remove everything under refs/pull-requests/ that match:
merge-basemerge-conflictedmerge-cleannotesto
Files should only be removed from the
/refs/pull-requests/…directories, and not the/stash-refs/pull-requestsdirectories.As always, please take a backup of the $
BITBUCKET_HOME before making any changes.
Was this helpful?