Caches

Bitbucket Pipelines supports caching build dependencies and directories, enabling faster builds and reducing the number of consumed build minutes.

What is dependency caching?

Most builds start by running commands that download dependencies from the internet, which can take a lot of time for each build. As the majority of dependencies stay the same, rather than download them every time, we recommend downloading them once into a cache which you can reuse for later builds.

Set up caching

To enable caching, add a caches section to your step .

Here's an example of how to cache your node_modules directory for a Node.js project using a pre-defined cache.

1 2 3 4 5 6 7 8 9 10 bitbucket-pipelines.yml pipelines: default: - step: caches: - node  script: - npm install - npm test

The first time this pipeline runs it won't find the node cache and so the npm commands will download dependencies from the internet. For future builds, our service will have them cached, so loading them into your build will be quicker.

Pre-defined caches

Pipelines provides pre-defined cache locations for commonly used language tools:

Cache name

In your step

Directories

docker

1 2 caches: - docker

n/a -  we create the cache from the layers generated by your build

composer

1 2 caches: - composer

~/.composer/cache

dotnetcore

1 2 caches: - dotnetcore

~/.nuget/packages

gradle

1 2 caches: - gradle

~/.gradle/caches

ivy2

1 2 caches: - ivy2

~/.ivy2/cache

maven

1 2 caches: - maven

~/.m2/repository

node

1 2 caches: - node

node_modules

pip

1 2 caches: - pip

~/.cache/pip

sbt

Note: When you use the sbt tool you need to enable both the sbt and ivy2 caches.

1 2 3 caches: - sbt - ivy2

~/.sbt

~/.ivy2/cache

Pre-defined caches don't support file-based cache keys. To better manage cached content, define a custom cache with file-based cache keys.

Custom caches for other build tools and directories

If your build tool isn't listed above, you can still define a custom cache for your repository in your bitbucket-pipelines.yml file.

Custom caches can support file-based cache keys. File-based cache keys allow for the generation and restoration of caches based on a set of files. Any changes to those files would result in a new cache. A typical use case might involve defining a cache key based on files that define a project's dependencies. When no dependencies have changed, Pipelines can restore cached dependencies from the cache. When dependencies are updated, the contents of the cache key files changes and Pipelines creates a new version of the cache.

  • First, in the definitions section of the .yml file, define the cache name, so you can then refer to that cache in any step.

  • Then define a set of files you want Pipelines to check for changes when deciding whether to use the cache or not, this is referred to as the key.

    • If the contents of the files have changed it indicates that new or updated dependencies are being used and Pipelines will download fresh dependencies.

  • Finally, specify the directory to cache using the path property.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 definitions: caches: my-bundler-cache: key: files: # the files you want pipelines to check for changes when deciding whether to use the cache, or download fresh dependencies. - Gemfile.lock - "**/*.gemspec" # glob patterns are supported for cache key files path: vendor/bundle # the directory you want to be cached. pipelines: default: - step: caches: - my-bundler-cache # Cache is defined above in the definitions section script: - bundle install --path vendor/bundle - ruby -e 'print "Hello, World\n"'

The defined cache will be saved after the first successful build.

Cache directory paths can be absolute, or relative to the clone directory, for example:

  • $HOME/.npm

  • ~/.gradle/wrapper

  • /usr/local/lib/node_modules

  • vendor/bundle

Note that for Ruby, you should install gems in a separate location from the system's gem repository, like the following:

1 $ bundle install --path vendor/bundle

Bypassing Cache invalidation logic

If you do not wish to use the automatic cache invalidation logic and simply want to cache a static directory for reuse every time, you can simply specify the name of the cache, followed by the directory you want to cache. Note: We do not recommend static directory caching unless you are experiencing issues with the automatic invalidation process.

1 2 3 4 5 6 7 8 9 10 11 definitions: caches: bundler: vendor/bundle pipelines: default: - step: caches: - bundler #cache is defined above in the definitions section script: - bundle install --path vendor/bundle

Caching with file-based cache keys

Custom caches can support file-based cache keys as an alternative to the basic `cache-name: /path` configuration. File-based cache keys allow for the generation and restoration of caches based on a set of files. Any changes to those files would result in a new cache.

A typical use case might involve defining a cache key based on files that define a project's dependencies. Pipelines can restore cached dependencies from the cache to avoid retrieving them from external sources and save time during builds. When dependencies are updated, the hash of the cache key files changes, and Pipelines creates a new version of the cache.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 definitions: caches: my-bundler-cache: key: files: - Gemfile.lock - "**/*.gemspec" # glob patterns are supported for cache key files path: vendor/bundle pipelines: default: - step: caches: - my-bundler-cache # Cache is defined above in the definitions section script: - bundle install --path vendor/bundle - ruby -e 'print "Hello, World\n"'

Caching multiple directories

Some builds might benefit from caching multiple directories. Simply reference multiple caches in your step like this:

1 2 3 4 5 6 7 8 9 10 11 12 image: openjdk:8 pipelines: default: - step: caches: - gradle # pre-defined cache - gradlewrapper # custom cache that must be defined below script: - ./gradlew build definitions: caches: gradlewrapper: ~/.gradle/wrapper

How does caching work?

When does a cache get saved?

Caches are saved on successful builds when the cache is empty. Only caches under 1GB once compressed are saved.

For the cache to compress to under 1GB, the size of the original images in the docker daemon must be < 2GB.
You can check the size by adding this command to the script in your bitbucket-pipelines.yml:

1 docker image inspect $(docker image ls -aq) --format {{.Size}} | awk '{totalSizeInBytes += $0} END {print totalSizeInBytes}'

When does a cache get restored?

At the beginning of each build, saved caches are downloaded if available and placed in their directories.

When does a cache get cleared?

Any cache which is older than 1 week will be cleared automatically and repopulated during the next build. Additionally, caches can be cleared manually in the Bitbucket UI. Just choose Caches at the top right of the Pipelines page, then delete a cache from the list using the trash icon:

Best practices

Dependency caching is all about reducing build times by avoiding downloading and building the same files repeatedly.

That means you should cache things like:

  • Language-specific dependencies.

  • Binaries of dependencies that take a while to compile.

You shouldn't cache:

  • Sensitive data such as passwords and credentials

Note: We do not cache symlinks, so if your dependencies rely heavily on them caching may not be efficient.

Remember that caches are temporary, so your builds should be configured to work whether or not the cache is present.

Additional Help