Bitbucket Cloud: Resolving Docker Cache Upload Failure in Parallel Pipeline Steps
Platform Notice: Cloud Only - This article only applies to Atlassian products on the cloud platform.
Summary
While running parallel pipeline steps with Docker build cache enabled, one may encounter the following Docker cache error:
Pipeline Build teardown
1
Cache "docker: docker.tar": Upload failedCause
This error typically occurs when parallel pipeline steps build different Docker images but save them to a common file named 'docker.tar' in the Docker cache. Each parallel step may overwrite the cache, leading to conflicts and upload failures.
Example:
sample bitbucket-pipelines.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
pipelines:
default:
- parallel:
- step:
name: 'Build 1'
caches:
- docker
services:
- docker
script:
- docker build .
- step:
name: 'Build 2'
caches:
- docker
services:
- docker
script:
- docker build .
- step:
name: 'Build 3'
caches:
- docker
services:
- docker
script:
- docker build .The "upload failed" error indicates a cache upload failure due to an existing cache, and caches are cleared only when they are more than a week old, as per the pipelines cache documentation.
Solution
To resolve this issue, create separate custom caches for each Docker image to store information. Manually configure the Docker cache to save in your image. Below is an example configuration for handling multiple Docker caches:
custom bitbucket-pipelines.yml with multiple docker caches
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
image: atlassian/default-image:3
definitions:
services:
docker:
memory: 1024
caches:
firstimage: firstimage
secondimage: secondimage
thirdimage: thirdimage
pipelines:
default:
- parallel:
- step:
name: 'Build 1'
caches:
- firstimage
services:
- docker
script:
- docker load -i firstimage/* || echo "No cache"
- docker build .
- mkdir -p firstimage && docker save $(docker images -aq) -o firstimage/cache.tar
- docker image ls -aq
- step:
name: 'Build 2'
caches:
- secondimage
services:
- docker
script:
- docker load -i secondimage/* || echo "No cache"
- docker build .
- mkdir -p secondimage && docker save $(docker images -aq) -o secondimage/cache.tar
- docker image ls -aq
- step:
name: 'Build 3'
caches:
- thirdimage
services:
- docker
script:
- docker load -i thirdimage/* || echo "No cache"
- docker build .
- mkdir -p thirdimage && docker save $(docker images -aq) -o thirdimage/cache.tar
- docker image ls -aqIn this example, each step uploads a specific Docker cache, avoiding conflicts and ensuring successful cache uploads.
Was this helpful?