Moving Git large files to Git LFS in Bitbucket Server

Summary

Using Git LFS to migrate Git large files to Git LFS.

Environment

Bitbucket Server and Data Center

Diagnosis

Repository has large files (more than 100MB) and it causes git clone and git push operations taking more time to complete.

Cause

• Begin tracking large files using Git LFS

• Reduce repository size

• Stop exponential repository size growth

Solution

All conversion should be done with a backup and on a testing environment. This is only intended for educational purposes and not covered in our support scope. Please consult an Atlassian Partners if more help is needed on this.

There are two options that you can choose in migrating large files to LFS as shown below.

1. Convert current and new big files to LFS and leave history untouched

• Performing these actions will not shrink Git repository size but it will stop high rate of repository size growth. It migrates the changes to current branch only. All other branches need to get the changes from current branch.

1. Archive the repository in Bitbucket application by renaming the repository (for eg. Project1 to Archived_Project1) - this is to ensure that we have the original repository if any error occurred during the process of enabling LFS tracking to large files

2. Create a new repository with Allow LFS setting

3. Clone the repository to a new temporary directory and checkout all branches and fetch all tags

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Clone the archived directory into a temporary directory in your local machine git clone <URL_to_archived_repo> tempdir To see a list of the different branches in archived repository do: git branch -a Checkout all the branches that you want to copy from Archived Repo to NEW Repo using: git checkout branch-name Now fetch all the tags from Archived Repo using: git fetch --tags Before doing the next step make sure to check your local tags and branches using the following commands: git tag git branch -a Now clear the link to the Archived repository with the following command: git remote rm origin

4. Setup LFS tracking on large files on currently checkout branch

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 Install Git LFS on your local machine using your favorite package manager $ git lfs install Git LFS initialized. You'll need to tell Git LFS to track the large binary file by specifying a pattern using the git lfs track command. For this example, we are tracking all files with .img format, please change it according to your requirement. $ git lfs track "*.img" After running git lfs track, you'll notice a new file named .gitattributes in the directory you ran the command from. .gitattributes is a Git mechanism for binding special behaviors to certain file patterns. Git LFS automatically creates or updates .gitattributes files to bind tracked file patterns to the Git LFS filter. However, you will need to commit any changes to the .gitattributes file to your repository yourself: $ git add .gitattributes $ git diff --cached $ git commit -m "Tracking .img files" Display a list of all patterns that are currently tracked by Git LFS (and the .gitattributes files they are defined in) by invoking git lfs track with no arguments: $ git lfs track

5. Push the archived repository to the newly created repository

   1 2 3 4 5 6 7 8 9 10 11 12 13 Now link your local repository to your newly created NEW repository using the following command: $ git remote add origin <url to NEW repo> Now add all files clones from the Archived Repo $ git add --all $ git commit -m "Adding all files cloned from old repository" Now push all your branches and tags with these commands: $ git push origin --all $ git push --tags If you have added files tracked by Git LFS, you will see some additional output from git push as the Git LFS content is transferred to the server.

6. Step 1-5 will only commit changes to one branch (master if you switched to it). You can merge changes from master to all branches using rebase or merge such as the example below.

   1 2 3 4 # For example, you can perform a rebase to a branch $ git checkout branch-name $ git rebase master $ git push origin --all

   If you have a lot of branches, you can perform a loop rebase or merge.

   1 $ for BRANCH in `ls .git/refs/heads`; do git rebase master $BRANCH; git push origin --all; done

2. Rewrite commit history using git lfs migrate

• This does shrink the Git repository size but since the migrated files (blobs in Git) are changed from the file content to an LFS pointer, you end up with new commits being created and thus new commit hashes. These new commits does not replace the old ones, but it is only creating an alternate history. This method also applies the migration changes to all branches.

1. Archive the repository in Bitbucket application by renaming the repository (for eg. Project1 to Archived_Project1) - this is to ensure that we have the original repository if any error occurred during the process of enabling LFS tracking to large files

2. Create a new repository with Allow LFS setting

3. Clone the repository to a new temporary directory and checkout all branches and fetch all tags

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Clone the archived directory into a temporary directory in your local machine git clone <URL_to_archived_repo> tempdir To see a list of the different branches in archived repository do: git branch -a Checkout all the branches that you want to copy from Archived Repo to NEW Repo using: git checkout branch-name Now fetch all the tags from Archived Repo using: git fetch --tags Before doing the next step make sure to check your local tags and branches using the following commands: git tag git branch -a Now clear the link to the Archived repository with the following command: git remote rm origin

4. Setup LFS tracking on large files

   1 2 3 4 5 6 7 8 9 Install Git LFS latest version on your local machine using your favorite package manager $ git lfs install Git LFS initialized. You'll need to tell Git LFS to track the large binary file by specifying a pattern using the git lfs track command. For this example, we are tracking all files with .img format, please change it according to your requirement. $ git lfs track "*.img" After running git lfs track, you'll notice a new file named .gitattributes in the directory you ran the command from. $ git add .gitattributes

5. Migrate Git history to Git LFS 

   1 2 3 4 5 # Dry run of your migration git lfs migrate info --everything --include="*.img" # Perform migration. Use the option --everything to perform a migration in all available branches. Use --include option to only migrate files you added for git lfs track. git lfs migrate import --everything --include="*.img" --verbose

6. Push the converted files to the new repository:

   1 2 3 4 Now link your local repository to your newly created NEW repository using the following command: $ git remote add origin <url to NEW repo> $ git push --tags $ git push origin --all

Note: If you would like to store Git LFS object files to Artifactory Repository. You can do that by adding the .lfsconfig file in the root of your project just before the push command in both Solution 1 & 2. If the file doesn't exist, create it.