How to migrate existing Bamboo artifacts to an AWS S3 bucket or to an SFTP server

Summary

This guide will show how to migrate your existing Bamboo artifacts to AWS S3 Bucket or to an SFTP server. This is particularly useful when your Bamboo server is running out of disk space and you are looking to reduce the impact on operations.

Environment

• Bamboo server with access to AWS S3 buckets or an SFTP server

• Locally stored artifacts of BambooRemoteArtifactHandler or ServerLocalArtifactHandler types

Solution

Identify and take notes of the artifacts you wish to move

You need to identify the artifacts you wish to move. Locate the artifacts, and prepare for the copy and updated SQL statements beforehand. There may also be cases where you'd rather move everything to the new location, in this case, there is no need to follow the provided steps to identify the artifacts below.

• Run a SQL select to link each artifact with its respective build located on the following KB article Disk space hotspots and cleanup best practices in Bamboo

  • The provided query will return the build key, size and location of an artifact folder

  • You may have to adapt it to your own requirements to have more/less information

• Artifacts are located on:

  • Bamboo 7 and earlier: <bamboo-home>/artifacts

  • Bamboo 8 and later: <bamboo-home>/shared/artifacts (replace <bamboo-home>/shared with your <bamboo-shared-home> if running on a cluster with customer configuration)

Moving to an S3 bucket

1. Go to "Bamboo Administration" >> "Artifact Handlers"

2. Configure the S3 Artifact handler Bucket settings; you can choose to:

   1. Use the same AWS credentials as the Elastic Bamboo (EC2)

   2. Specify different AWS credentials

3. Copy the artifacts manually to the S3 bucket's root folder. We are not going to get into the details of the copy process as this varies from customer to customer

4. When copying the files, make sure to copy the whole folder names recursively such as "plan-1234567/*" to the S3 bucket's parent path

5. Shutdown your Bamboo Instance and take a full database backup, including <bamboo-home>, <bamboo-install> and the Database

6. Modify the database so Bamboo will reference the artifacts on the S3 bucket:

   Database Update Query

   update artifact set link_type = 'com.atlassian.bamboo.plugin.artifact.handler.remote:S3ArtifactHandler' where storage_tag = '<storage_tag>';

7. Start Bamboo and disable the old Artifact handler (Agent-local artifact handler, Bamboo server artifact handler)

8. Try to access the artifacts from a build summary-> Artifacts tab. If you have the required AWS S3 bucket permissions, Bamboo can access the artifacts in the S3 bucket.

9. Monitor the <bamboo-home>/logs/atlassian-bamboo.log file for any errors while accessing the artifacts.

10. If you see errors or have trouble accessing the migrated artifacts, please contact Atlassian Support.

Moving to an SFTP server

1. Configure the SFTP artifact handler

2. Copy the artifacts manually to the SFTP remote location defined on the SFTP Artifact handler. We are not going to get into the details of the copy process as this varies from customer to customer

3. When copying the files, make sure to copy the whole folder names recursively, such as "plan-1234567/*" to the SFTP server configured location

4. Shutdown your Bamboo Instance and take a full database backup, including <bamboo-home>, <bamboo-install> and the Database

5. Update the database so Bamboo will reference the artifacts on the SFTP remote path:

   Database Update Query

   update artifact set link_type ='com.atlassian.bamboo.plugins.bamboo-scp.plugin:sftpArtifactHandler' where storage_tag = '<storage_tag>';

6. Start Bamboo and disable the old Artifact handler (Agent-local artifact handler, Bamboo server artifact handler)

7. Try to access the artifacts. If you have configured the required SFTP rights, Bamboo will be able to access the artifacts from the SFTP server

8. Monitor the <bamboo-home>/logs/atlassian-bamboo.log file for any errors while accessing the artifacts.

9. If you see errors or have trouble accessing the migrated artifacts, please contact Atlassian Support.

Related articles

• Disk space hotspots and cleanup best practices in Bamboo — the SQL query for identifying artifacts and their builds.

• Configuring Elastic Bamboo — AWS credential configuration (referenced inline)

• Configuring the SFTP artifact handler — a required prerequisite for the SFTP migration path