How to migrate existing Bamboo artifacts to an AWS S3 bucket or to an SFTP server

Summary

This guide will show how to migrate your existing Bamboo artifacts to AWS S3 Bucket or to an SFTP server. This is particularly useful when your Bamboo server is running out of disk space and you are looking to reduce the impact on operations.

Environment

• Bamboo server with access to AWS S3 buckets or an SFTP server

• Locally stored artifacts of BambooRemoteArtifactHandler or ServerLocalArtifactHandler types

Solution

Identify and take notes of the artifacts you wish to move

You need to identify the artifacts you wish to move. Locate the artifacts, and prepare for the copy and updated SQL statements beforehand. There may also be cases where you'd rather move everything to the new location, in this case, there is no need to follow the provided steps to identify the artifacts below.

• Run a SQL select to link each artifact with its respective build located on the following KB article Disk space hotspots and cleanup best practices in Bamboo

  • The provided query will return the build key, size and location of an artifact folder

  • You may have to adapt it to your own requirements to have more/less information

• Artifacts are located on:

  • Bamboo 7 and earlier: <bamboo-home>/artifacts

  • Bamboo 8 and later: <bamboo-home>/shared/artifacts (replace <bamboo-home>/shared with your <bamboo-shared-home> if running on a cluster with customer configuration)

Moving to an S3 bucket

1. Go to "Bamboo Administration" >> "Artifact Handlers"

2. Configure the S3 Artifact handler Bucket settings; you can choose to:

   1. Use the same AWS credentials as the Elastic Bamboo (EC2)

   2. Specify different AWS credentials

3. Copy the artifacts manually to the S3 bucket's root folder. We are not going to get into the details of the copy process as this varies from customer to customer

4. When copying the files, make sure to copy the whole folder names recursively such as "plan-1234567/*" to the S3 bucket's parent path

5. Shutdown your Bamboo Instance and take a full database backup, including <bamboo-home>, <bamboo-install> and the Database

6. Modify the database so Bamboo will reference the artifacts on the S3 bucket:

   Database Update Query

   1 2 3 update artifact set link_type = 'com.atlassian.bamboo.plugin.artifact.handler.remote:S3ArtifactHandler' where storage_tag = '<storage_tag>';

7. Start Bamboo and disable the old Artifact handler (Agent-local artifact handler, Bamboo server artifact handler)

8. Try to access the artifacts. If you have the required AWS S3 bucket rights Bamboo will be able to access the artifacts from the S3 bucket.

Moving to an SFTP server

1. Configure the SFTP artifact handler

2. Copy the artifacts manually to the SFTP remote location defined on the SFTP Artifact handler. We are not going to get into the details of the copy process as this varies from customer to customer

3. When copying the files, make sure to copy the whole folder names recursively such as "plan-1234567/*" to the SFTP server configured location

4. Shutdown your Bamboo Instance and take a full database backup, including <bamboo-home>, <bamboo-install> and the Database

5. Update the database so Bamboo will reference the artifacts on the SFTP remote path:

   Database Update Query

   1 2 3 update artifact set link_type ='com.atlassian.bamboo.plugins.bamboo-scp.plugin:sftpArtifactHandler' where storage_tag = '<storage_tag>';

6. Start Bamboo and disable the old Artifact handler (Agent-local artifact handler, Bamboo server artifact handler)

7. Try to access the artifacts. If you have configured the required SFTP rights Bamboo will be able to access the artifacts from the SFTP server