How to prevent agents from filling up with Docker cached images
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The steps outlined on this article are provided AS-IS. This means we've had reports of them working for some customers — under certain circumstances — yet are not officially supported, nor can we guarantee they'll work for your specific scenario.
You may follow through and validate them on your own non-prod environments prior to production or fall back to supported alternatives if they don't work out.
We also invite you to reach out to our Community for matters that fall beyond Atlassian's scope of support!
When running Docker tasks with Bamboo, the agents will cache any related Docker images used. Depending on the use-case scenario, the number of images cached can get quite large, and it can end up consuming lots of space. This article covers a way to clean up the agents.
Environment
Bamboo with Docker tasks or Docker Runner.
Solution
You can set up a Cron job to periodically clean up the images. The following bash script can be adapted to your environment and be used as a base for a clean-up solution:
1
docker rmi $(docker image ls | grep test-repo | sort -k2 -r | tail -n+2 | awk 'BEGIN{OFS=":"} {print $1,$2}')It does the following:
Gets a list of all Docker images available locally.
Filters out only the ones to clean up ("test-repo").
Sorts them by the tag version, so we have newer tags at the top.
Select all but the first result, so we don't remove the most recent image.
Forms an
image_name:tagstring, which is then deleted by thedocker rmicommand.
For example, let's say a remote agent has the following images locally, going from tag versions 1.1.18 to 1.1.24:
1
2
3
4
5
6
7
8
9
$ docker image ls | grep test-repo | sort -k2 -r
test/test-repo 1.1.24 18bc1b3b2db6 6 weeks ago 5.57MB
test/test-repo 1.1.23 18bc1b3b2db6 6 weeks ago 5.57MB
test/test-repo 1.1.22 18bc1b3b2db6 6 weeks ago 5.57MB
test/test-repo 1.1.21 18bc1b3b2db6 6 weeks ago 5.57MB
test/test-repo 1.1.20 18bc1b3b2db6 6 weeks ago 5.57MB
test/test-repo 1.1.19 18bc1b3b2db6 6 weeks ago 5.57MB
test/test-repo 1.1.18 18bc1b3b2db6 6 weeks ago 5.57MBBy running that command, it removes all images from 1.1.18 up to 1.1.23, leaving only the most recent one (1.1.24):
1
2
3
4
5
6
7
8
$ docker rmi $(docker image ls | grep test-repo | sort -k2 -r | tail -n+2 | awk 'BEGIN{OFS=":"} {print $1,$2}')
Untagged: test/test-repo:1.1.23
Untagged: test/test-repo:1.1.22
Untagged: test/test-repo:1.1.21
Untagged: test/test-repo:1.1.20
Untagged: test/test-repo:1.1.19
Untagged: test/test-repo:1.1.18We now only have the latest image cached locally:
1
2
3
$ docker image ls | grep test-repo
test/test-repo 1.1.24 18bc1b3b2db6 6 weeks ago 5.57MBAutomating the clean-up using a Bamboo Build Plan
Depending on the number of agents you have with a Docker capability set, it might be feasible to implement a cleanup script through Bamboo itself, using a new Build Plan. For example:
Create a new Build Plan to work as a "Docker image cleaner".
Add a Job with a Script Task containing the script mentioned above. ⚠️ You'll need to adapt it to your use-case scenario.
The tricky part is to make it run on all agents that have Docker. In order to do that, you can create a custom capability on those agents that have Docker and make it a requirement, having one Script Job for each agent. For example:
Create a custom capability and add it to all agents that have Docker installed. Each agent should have its own unique value for that capability.
You can clone the Script Job so we have one job for each build agent. In order to force each job to run on a specific agent, you can add a requirement to the job, looking for the custom capability with the agent's unique value:
In this case, Job 1 requires value 1, which runs on Agent A. Job 2 requires value 2, which runs on Agent B. And Job 3 requires value 3, which runs on Agent C.
You can then add a Scheduled Trigger to the cleanup build to run it on a schedule, according to your needs.
This is just a suggestion for implementing this through Bamboo, but it might not be the best one depending on the number of Docker agents you have, for which the Cron job would be the best approach.
Was this helpful?