Caches

Bitbucket Pipelines supports caching build dependencies and directories, enabling faster builds and reducing the number of consumed build minutes.

What is dependency caching?

Most builds start by running commands that download dependencies from the internet, which can take a lot of time for each build. As the majority of dependencies stay the same, rather than download them every time, we recommend downloading them once into a cache which you can reuse for later builds.

Set up caching

To enable caching, add a caches section to your step .

Here's an example of how to cache your node_modules directory for a Node.js project using a pre-defined cache.

bitbucket-pipelines.yml
        
pipelines:
  default:
    - step:
        caches:
          - node
        script:
          - npm install
          - npm test

The first time this pipeline runs it won't find the node cache and so the npm commands will download dependencies from the internet. For future builds, our service will have them cached, so loading them into your build will be quicker.

Pre-defined caches

Pipelines provides pre-defined cache locations for commonly used language tools:

Cache name	In your step	Directories
docker	`1 2caches: - docker`	n/a - we create the cache from the layers generated by your build
composer	`1 2caches: - composer`	~/.composer/cache
dotnetcore	`1 2caches: - dotnetcore`	~/.nuget/packages
gradle	`1 2caches: - gradle`	~/.gradle/caches
ivy2	`1 2caches: - ivy2`	~/.ivy2/cache
maven	`1 2caches: - maven`	~/.m2/repository
node	`1 2caches: - node`	node_modules
pip	`1 2caches: - pip`	~/.cache/pip
sbt Note: When you use the sbt tool you need to enable both the sbt and ivy2 caches.	`1 2 3caches: - sbt - ivy2`	~/.sbt ~/.ivy2/cache

Pre-defined caches don't support file-based cache keys. To better manage cached content, define a custom cache with file-based cache keys.

Custom caches for other build tools and directories

If your build tool isn't listed above, you can still define a custom cache for your repository in your bitbucket-pipelines.yml file.

Custom caches can support file-based cache keys. File-based cache keys allow for the generation and restoration of caches based on a set of files. Any changes to those files would result in a new cache. A typical use case might involve defining a cache key based on files that define a project's dependencies. When no dependencies have changed, Pipelines can restore cached dependencies from the cache. When dependencies are updated, the contents of the cache key files changes and Pipelines creates a new version of the cache.

First, in the definitions section of the .yml file, define the cache name, so you can then refer to that cache in any step.
Then define a set of files you want Pipelines to check for changes when deciding whether to use the cache or not, this is referred to as the key.
- If the contents of the files have changed it indicates that new or updated dependencies are being used and Pipelines will download fresh dependencies.
Finally, specify the directory to cache using the path property.

definitions:
  caches:
    my-bundler-cache:
      key:
        files: # the files you want pipelines to check for changes when deciding whether to use the cache, or download fresh dependencies.
          - Gemfile.lock
          - "**/*.gemspec" # glob patterns are supported for cache key files
      path: vendor/bundle # the directory you want to be cached.

pipelines:
  default:
    - step:
  caches:
    - my-bundler-cache # Cache is defined above in the definitions section
  script:
    - bundle install --path vendor/bundle
    - ruby -e 'print "Hello, World\n"'

The defined cache will be saved after the first successful build.

Cache directory paths can be absolute, or relative to the clone directory, for example:

$HOME/.npm
~/.gradle/wrapper
/usr/local/lib/node_modules
vendor/bundle

Note that for Ruby, you should install gems in a separate location from the system's gem repository, like the following:

$ bundle install --path vendor/bundle

Bypassing Cache invalidation logic

If you do not wish to use the automatic cache invalidation logic and simply want to cache a static directory for reuse every time, you can simply specify the name of the cache, followed by the directory you want to cache. Note: We do not recommend static directory caching unless you are experiencing issues with the automatic invalidation process.

definitions:
  caches:
    bundler: vendor/bundle
    
pipelines:
  default:
    - step:
  caches:
    - bundler #cache is defined above in the definitions section
  script:
    - bundle install --path vendor/bundle

Caching with file-based cache keys

Custom caches can support file-based cache keys as an alternative to the basic `cache-name: /path` configuration. File-based cache keys allow for the generation and restoration of caches based on a set of files. Any changes to those files would result in a new cache.

A typical use case might involve defining a cache key based on files that define a project's dependencies. Pipelines can restore cached dependencies from the cache to avoid retrieving them from external sources and save time during builds. When dependencies are updated, the hash of the cache key files changes, and Pipelines creates a new version of the cache.

definitions:
  caches:
    my-bundler-cache:
      key:
        files:
          - Gemfile.lock
          - "**/*.gemspec" # glob patterns are supported for cache key files
      path: vendor/bundle

pipelines:
  default:
    - step:
        caches:
          - my-bundler-cache # Cache is defined above in the definitions section
        script:
          - bundle install --path vendor/bundle
          - ruby -e 'print "Hello, World\n"'

Caching multiple directories

Some builds might benefit from caching multiple directories. Simply reference multiple caches in your step like this:

image: openjdk:8
pipelines:
  default:
    - step:
        caches:
          - gradle               # pre-defined cache
          - gradlewrapper        # custom cache that must be defined below
        script:
          - ./gradlew build
definitions:
  caches:
    gradlewrapper: ~/.gradle/wrapper 

How does caching work?

When does a cache get saved?

Caches are saved on successful builds when the cache is empty. Only caches under 1GB once compressed are saved.

For the cache to compress to under 1GB, the size of the original images in the docker daemon must be < 2GB.
You can check the size by adding this command to the script in your bitbucket-pipelines.yml:

docker image inspect $(docker image ls -aq) --format {{.Size}} | awk '{totalSizeInBytes += $0} END {print totalSizeInBytes}'

When does a cache get restored?

At the beginning of each build, saved caches are downloaded if available and placed in their directories.

When does a cache get cleared?

Any cache which is older than 1 week will be cleared automatically and repopulated during the next build. Additionally, caches can be cleared manually in the Bitbucket UI. Just choose Caches at the top right of the Pipelines page, then delete a cache from the list using the trash icon:

Best practices

Dependency caching is all about reducing build times by avoiding downloading and building the same files repeatedly.

That means you should cache things like:

Language-specific dependencies.
Binaries of dependencies that take a while to compile.

You shouldn't cache:

Sensitive data such as passwords and credentials

Note: We do not cache symlinks, so if your dependencies rely heavily on them caching may not be efficient.

Remember that caches are temporary, so your builds should be configured to work whether or not the cache is present.

Was this helpful?

It wasn't accurate

It wasn't clear

It wasn't relevant

Still need help?

The Atlassian Community is here for you.

Ask the Community