High load on Hazelcast and Bitbucket due to failing GC on refs/pull-requests/<pr id>/v* files
問題
High CPU and performance issues in Bitbucket appear along with the following messages in the atlassian-bitbucket-yyyy-mm-dd.log:
c.a.s.i.c.HazelcastBucketedExecutor Attempt 2 of 5 at processing bucket 'git:gc@385d519558f5df598f79' for executor 'git:gc' failed: ''/srv/git/current/bin/git gc' exited with code 255 saying: fatal: bad object refs/pull-requests/1/v3You may also see similar errors for the following fiile names:
merge-basemerge-conflictedmerge-cleannotesto
These could be accompanied by hundreds of Hazelcast errors in the logs as well.
原因
Prior to Stash 2.9 the refs/pull-requests/<pr id>/v* files were used, and from 2.9 onwards we use them from the stash-refs directory instead. It's possible the upgrade tasks failed to remove the v* files or these files were restored from a backup. Garbage collection continuously failing garbage collection can put a server under a huge amount of load.
Bitbucket 5+ moves GC off the BucketedExecutor and Hazelcast. Garbage collection will only create load on HazelCast in Bitbucket 4.x versions.
ソリューション
Remove everything under refs/pull-requests/ that match:
merge-basemerge-conflictedmerge-cleannotesto
- Files should only be removed from the
/refs/pull-requests/…directories, and not the/stash-refs/pull-requestsdirectories. - As always, please take a backup of the $
BITBUCKET_HOME before making any changes.