Caches

 

Bitbucket Pipelines is able to cache external build dependencies and directories, such as 3rd-party libraries, between builds providing faster builds, and reducing the number of consumed build minutes.

What is dependency caching?

Most builds start by running commands that download dependencies from the internet, which can take a lot of time for each build. As the majority of dependencies stay the same, rather than download them every time, we recommend downloading them once into a cache which you can reuse for later builds.

Set up caching

To enable caching, add a caches section to your step .

Here's an example of how to cache your node_modules directory for a Node.js project using a pre-defined cache.

1 2 3 4 5 6 7 8 9 10 bitbucket-pipelines.yml pipelines: default: - step: caches: - node  script: - npm install - npm test

The first time this pipeline runs it won't find the node cache and so the npm commands will download dependencies from the internet. For future builds, our service will have them cached, so loading them into your build will be quicker.

Pre-defined caches

Pipelines provides pre-defined cache locations for commonly used language tools:

Cache name

In your step

Directories

docker

1 2 caches: - docker

n/a -  we create the cache from the layers generated by your build

composer

1 2 caches: - composer

~/.composer/cache

dotnetcore

1 2 caches: - dotnetcore

~/.nuget/packages

gradle

1 2 caches: - gradle

~/.gradle/caches

ivy2

1 2 caches: - ivy2

~/.ivy2/cache

maven

1 2 caches: - maven

~/.m2/repository

node

1 2 caches: - node

node_modules

pip

1 2 caches: - pip

~/.cache/pip

sbt

1 2 3 caches: - sbt - ivy2

~/.sbt

~/.ivy2/cache

Note that when you use the sbt tool you need to enable both the sbt and ivy2 caches.

Custom caches for other build tools and directories

If your build tool isn't listed above, you can still define a custom cache for your repository in your bitbucket-pipelines.yml file. First, in the definitions section of the yml, define the cache name and the directory to be cached, then you can refer to that cache in any step:

1 2 3 4 5 6 7 8 9 10 11 12 image: ruby:2.4.0 pipelines: default: - step: caches: - bundler #cache is defined below in the definitions section script: - bundle install --path vendor/bundle definitions: caches: bundler: vendor/bundle

The defined cache will be saved after the first successful build.

Cache directory paths can be absolute, or relative to the clone directory, for example:

  • $HOME/.npm

  • ~/.gradle/wrapper

  • /usr/local/lib/node_modules

  • vendor/bundle

Note that, for Ruby, you should install gems in a separate location from the system's gem repo, like this:

 

1 $ bundle install --path vendor/bundle

Caching multiple directories

Some builds might benefit from caching multiple directories. Simply reference multiple caches in your step like this:

1 2 3 4 5 6 7 8 9 10 11 12 image: openjdk:8 pipelines: default: - step: caches: - gradle # pre-defined cache - gradlewrapper # custom cache that must be defined below script: - ./gradlew build definitions: caches: gradlewrapper: ~/.gradle/wrapper

How does caching work?

When does a cache get saved?

Caches are saved on successful builds when the cache is empty. Only caches under 1GB once compressed are saved.

For the cache to compress to under 1GB, the size of the original images in the docker daemon must be < 2GB.
You can check the size by adding this command to the script in your bitbucket-pipelines.yml:

docker image inspect $(docker image ls -aq) --format {{.Size}} | awk '{totalSizeInBytes += $0} END {print totalSizeInBytes}'

When does a cache get restored?

At the beginning of each build, saved caches are downloaded if available and placed in their directories.

When does a cache get cleared?

Any cache which is older than 1 week will be cleared automatically and repopulated during the next build. Additionally, caches can be cleared manually in the Bitbucket UI. Just choose Caches at the top right of the Pipelines page, then delete a cache from the list using the trash icon:

Best practices

Dependency caching is all about reducing build times by avoiding downloading and building the same files repeatedly.

That means you should cache things like:

  • Language-specific dependencies.

  • Binaries of dependencies that take a while to compile.

You shouldn't cache:

  • Sensitive data such as passwords and credentials

Note: We do not cache symlinks, so if your dependencies rely heavily on them caching may not be efficient.

Remember that caches are temporary, so your builds should be configured to work whether or not the cache is present.

Additional Help

Ask the Community