Git operations fail due to "git-pack-objects died with error"
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Git commands don't complete successfully.
The following appears in the command console output:
1
error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server ErrorThe following appears in the atlassian-bitbucket.log
1
2
3
4
ERROR [http-nio-7990-exec-52] <username> @1WU14K3x805x122891x1 193.39.66.214 "POST /scm/<project_key>/<repository_slug>.git/git-upload-pack HTTP/1.1" c.a.s.i.s.g.p.h.GitSmartExitHandler <project_key>/<repository_slug>[10]: Read request from 193.39.66.214 failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1
The following was written to stderr:
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.Cause
There may be several causes for this problem:
Cause #1 - Corrupted Git repository due to filesystem corrupton
The repository has become corrupted as a consequence of a file system corruption.
Note: cloning the repository directly on the file system (without using Bitbucket Data Center) may still succeed since a Git clone does not run any additional checks and assumes that the repository is in a consistent state.
Cause #2 - Corrupted Git repository due to insufficient memory
The repository has become corrupted after a repack operation failed due to insufficient memory. Attempting a manual repack of the repository (see Workaround for Cause #1) shows the following error.
1
2
Delta compression using up to 4 threads
error: pack-objects died of signal 9)Workaround
If the steps below don't resolve the problem, the repository may be not recoverable. In this case, the situation can be resolved by restoring a backup generated prior to the corruption.
Cause #1
Manually repack the repository.
Stop Bitbucket Server
Copy the $BITBUCKET_HOME/shared/data/repositories/<repository_id> folder. This is to make sure that a backup of the entire repository is available.
Run the following commands from the
$BITBUCKET_HOME/shared/data/repositories/<repository_id>path:1 2git fsck --no-dangling --name-objects git repack -adfln --keep-unreachable --depth=20 --window=200Start Bitbucket Server
Cause #2
Set a limit for the memory allocated when running the repack.
Stop Bitbucket Server
Copy the $BITBUCKET_HOME/shared/data/repositories/<repository_id> folder. This is to make sure that a backup of the entire repository is available.
Navigate to the
$BITBUCKET_HOME/shared/data/repositories/<repository_id>path.Add --window-memory=<n> to the git repack command and see if adjusting the value produces a different output and allows the repack to complete successfully.
1
2
3
4
git fsck --no-dangling --name-objects
git repack -adfln --keep-unreachable --depth=20 --window=200 --window-memory=500m
..
git repack -adfln --keep-unreachable --depth=20 --window=200 --window-memory=1gStart Bitbucket Server
Was this helpful?