Pipelines default docker cache not working for docker build steps

Summary

When using Bitbucket Pipelines, you may encounter an issue where the Docker cache does not work as expected for Docker build steps. Although the build setup steps indicate that the Docker cache has been downloaded, it is not being used in Docker build steps.

As a result, Docker build steps download build dependencies for each layer, significantly slowing down the build process. Interestingly, the same Docker steps run locally utilize Docker cache for build dependencies and run faster.

Solution

There are several potential reasons for this problem, but one of the common solutions is not to use "DOCKER_BUILDKIT=1" in the build step. According to the Bitbucket Pipeline documentation, enabling DOCKER_BUILDKIT=1 can affect caching behavior. When DOCKER_BUILDKIT is enabled, caching is not shared across different builds, and it's limited to the build running on the same Docker node where the build is executed. This means that caching for subsequent steps in Pipelines is unsupported for BuildKit builds; caching only works on the same Docker node that created the cache.

To address this issue, consider the following options:

• Disable DOCKER_BUILDKIT: By default, DOCKER_BUILDKIT is enabled in your Pipelines. you can choose to disable it your Pipelines configuration by using DOCKER_BUILDKIT=0. This way, you can continue to use default pipelines docker caching in your Pipelines which should work without any issues.

• Self-Hosted Runners: Another option is to self-host your builds with runners. By using your own runners, you have more control over the build environment, including caching. You can implement remote caching, as described in Bitbucket's blog post on faster-ci-builds-with-docker-remote-caching.

By considering these solutions and options, you can work towards resolving the Docker cache issue in Bitbucket Pipelines and improve your build times. we do have a feature request BCLOUD-22592 - Support caching layers produced in Docker BuildKit build on Bitbucket's public issue tracker to address the Docker cache behavior issue with docker BuildKit enabled. If none of the above solutions work, please reach out to Bitbucket Support for further assistance.