Git LFS pull hangs indefinitely in Bamboo with Git LFS 2.6.0
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Git LFS pull hangs indefinitely in Bamboo when the repository has > ~100 LFS objects and the build eventually fails after reaching the Git command timeout (default 180 minutes).
Diagnosis
Environment
Git LFS 2.6.0 client is installed on the Bamboo Server (local agent) or Remote Agent performing the build.
Diagnostic Steps
With verbose Git logging enabled in Bamboo: Edit Repository >> Advanced Options >> Enable verbose logging the build hangs at the below line in the build log:
1
2
3
4
simple 29-Nov-2018 17:32:16 /usr/bin/git remote set-url origin ssh://d0cbc387-4ab8-444a-81bd-19700ad67444@127.0.0.1:46388/lfs/lfsrepotest.git
simple 29-Nov-2018 17:32:16 /usr/bin/git lfs pull
command 29-Nov-2018 17:39:12 Request to stop 'LFS-LFS2-JOB1-5' received from charlie
simple 29-Nov-2018 17:39:12 /usr/bin/git remote set-url origin file:///var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/e8904ea3b8469d750cc8e102ce3b1f91ae7fb3bastrace -p <pid> of the Git LFS pull shows the process is stuck in a FUTEX_WAIT loop:
1
2
3
4
[pid 2331] 17:34:04.472561 nanosleep({0, 20000}, NULL) = 0
[pid 2331] 17:34:04.472903 futex(0xce2530, FUTEX_WAIT_PRIVATE, 0, {60, 0}) = -1 ETIMEDOUT (Connection timed out)
[pid 2331] 17:35:04.473506 nanosleep({0, 20000}, NULL) = 0
[pid 2331] 17:35:04.473956 futex(0xce2530, FUTEX_WAIT_PRIVATE, 0, {60, 0}Cause
There's an issue with Git LFS 2.6.0 that causes problems when pulling from a clone with Git alternates --shared or --reference and this is how Bamboo leverages benefits of it's local Git cache:
Solution
Workaround
If using remote agents, you can disable repository caching at Edit Repository >> Advanced Options >> Enable repository caching on remote agents which will workaround the issue (not recommend as the repository cache is very beneficial to speed up repository operations, reducing load on your repository server and network).
Resolution
Upgrade to a fixed version of Git LFS, 2.6.1 and above: https://github.com/git-lfs/git-lfs/releases/tag/v2.6.1
Was this helpful?