Infrastructure changes in Bitbucket Pipelines

This page tracks internal infrastructure changes to Bitbucket Pipelines that in rare cases might affect customer builds.

May 2021 - Flatcar OS upgrade rollback

After the upgrade to Kubernetes FlatcarOS v2765.2.2, we detected errors with Ruby/Rails failing with ActiveStorage::IntegrityError and yarn builds hanging. Because of this, we are rolling back to Kubernetes FlatcarOS v2605.12.0.

May 2021 - Flatcar OS upgrade

Starting on May 4th, we will be upgrading our Kubernetes FlatcarOS to v2765.2.2 for all customers. This change should be transparent to users.

March 2021 - containerd minor version upgrade

Starting on March 29th, we will be upgrading containerd on the Kubernetes nodes from v1.3.9 to v1.4.3. Within this latest version upgrade, containerd ended support for certain versions of some old images, like python:3.5.1 and redis:2.8.23. Update to the latest version to fix any issues.

March 2021 - Kubernetes cluster upgrade

Starting on March 1st, we will be upgrading our Kubernetes cluster from v1.18.0 to v1.19.7 for all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

February 2021 - Runtime environment migration to Kata Container

Starting on March 1st, we will be rolling out the migration from Docker & Containerd to Kata Container lightweight virtual machines for increased isolation and security of customer workloads to our Standard and Premium Bitbucket Cloud customers. Any users on a Free plan have already been migrated. This change should be transparent to users.

February 2021 - Flatcar OS upgrade

Starting on February 17th, we will be upgrading our Kubernetes FlatcarOS to v2605.12.0 for all customers. This change should be transparent to users.

February 2021 - Git LFS client upgrade

As of February 2021, we upgraded the Git LFS client from version 2.6.0 to 2.13.2. This change should be transparent to users.

February 2021 - Changes in docker’s daemon

Starting on February 3rd, we will be incrementally rolling out an upgraded version from 19.03.13 to 19.03.15.

January 2021- FlatcarOS upgrade

Starting on January 27th, we will be upgrading our Kubernetes FlatcarOS to v2605.10.0 for all customers. This change should be transparent to users.

January 2021 - containerd minor version upgrade

Starting on January 20th, we will be upgrading containerd on the Kubernetes nodes from v1.3.4 to v1.3.9.

January 2021 - Kubernetes cluster upgrade

Starting on January 20th, we will be upgrading our Kubernetes cluster from v1.17.9 to v1.18.10 for all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

December 2020 - Kubernetes cluster added

We’ve added a new Kubernetes cluster to run builds across some pipelines. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

November 2020 - Kubernetes node OS upgrade

Starting on November 11th, we will be rolling out an upgraded version of the operating system that our Kubernetes build nodes run. This involves a change of Linux kernel versions from 4.19.143 to 5.4.72. This change should be transparent to users.

November 2020 - Changes in docker’s daemon

Starting on November 2nd through November 10th, we will be incrementally rolling out an upgraded version from 18.09.9 to 19.03.13.

September 2020 - Changes in docker’s daemon

The docker daemon version used in the docker service has been upgraded from 18.09.1 to the latest 18.09.9

September 2020 - changes to docker in docker container filesystem permissions

Starting on 2nd September the docker in docker container's root filesystem will be read-only except for the directories required to use docker in order to harden security. The BITBUCKET_CLONE_DIR and its subdirectories, any other volumes you create, will remain writable.

April 2020 - containerd minor version upgrade

Starting on 22rd April containerd on the kubernetes nodes was upgraded from 1.2.x to 1.3.4.

Starting on 29th April switch to containerd-shim-runc-v2 to use per-pod shim instead of per-container shim.

November 2019  - Kubernetes cluster upgrades

Starting on 20th November, we will be progressively rolling out Kubernetes cluster upgrades to all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

August 2019  - containerd runtime changes on nodes

Starting from August 7th, the nodes in our kubernetes build cluster will be changed to run with containerd rather than docker. The change should be transparent to most Bitbucket Pipelines users.
Update August 9th: We've identified 2 issues with the rollout:

• Users with docker images hosted on outdated Sonatype Nexus instances will have trouble pulling images via  containerd. Please file a support ticket at https://support.atlassian.com/contact to be excluded from the migration and consider upgrading the Nexus instance (see https://issues.sonatype.org/browse/NEXUS-12684).

• Users with private docker images hosted on bintray.com will experience 401 Unauthorized response with containerd. We've identified the issue to be with the provider and are communicating with them on resolution. Please file a support ticket to be excluded from the containerd until the issue is resolved using the following link https://support.atlassian.com/contact.

April 2019 - SSH keyscan performed from build environment

Starting from the 17th of April 2019, SSH keyscans will be performed from within the build environment. This means you will need to add the valid IP addresses to an allowlist for Bitbucket Pipelines build environments to continue using this. The IP addresses in use by Bitbucket Pipelines services will no longer need to be allowlisted.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

October 2018 - New outbound IP addresses

Starting from the 8th of November 2018, new IP addresses will be in use by Bitbucket Pipelines services (not our build infrastructure).

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

September 2018 - Docker in Docker User Namespace Remapping

On 27th September 2018, we rolled out a change to enable user namespace remapping in our docker in docker daemon we provide to users as part of a steps execution to harden the security of pipelines.

September 2018 - New outbound IP addresses

On 12th September 2018, new IP addresses were provisioned for our build infrastructure to offer future multi-region failover in our Kubernetes infrastructure. These addresses will become active in the next two weeks.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

August 2018 - Performance improvements by moving to EC2 M5d instance types

On 1st August 2018, we swapped our Kubernetes nodes from using EC2 M4 instance types to M5d's. M5d instances use NVMe drives (instead of EBS volumes), which are much faster, as well as located on the underlying compute hardware, not having then the overhead of transferring data on the drives over a storage network.

March 2018 - New outbound IP addresses

On 15th March 2018, new IP addresses were provisioned for our build infrastructure. These addresses will become active in the next two weeks.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

November 2017 - Docker now a service, limited to 1 GB memory

On 28 November 2017, as part of implementing docker-run support in Pipelines, we now treat Docker as a Pipelines service. This means commands executed via Docker will have a memory limit of 1 GB, and builds that enable Docker can only use two additional services per build step.

There are a very small number of existing builds that use three services and have Docker enabled that will break with this change. We have directly notified customers who have recently run builds with this configuration.

Our recommendation is to either stop running one of your services or change one service to run using "docker run" instead (YAML example). Docker run support will also give you the flexibility to start multiple Docker containers in the same build, including via docker-compose files.

October 2017 - New outbound IP addresses

On 25 October 2017, new IP addresses were provisioned for our build infrastructure.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

September 2017 - Docker upgrade

On 7 September 2017, we upgraded the Docker daemon provided to Pipelines build containers, from 1.12.6 to 17.05.

Please see this ticket for more details: https://bitbucket.org/site/master/issues/14333/upgrade-docker-for-multi-stage-builds

February 2017 - New infrastructure

As of February 2017, we're rolling out changes to Pipeline's build infrastructure to provide a foundation for upcoming new features. Pipelines still executes your scripts in an isolated Docker container, and most people won't notice any change in behavior.

There are a couple of minor differences that may affect some people, described below.

How to tell if you have the new infrastructure

You can tell if you've got the updated infrastructure by looking at the log file. The 'Build setup' section at the top will be noticeably shorter, and will no longer contain docker run commands.

You still have the old infrastructure if you see docker run commands in the 'Build setup' section of your log file similar to the following:

What's changed?

Scripts are no longer run in an interactive shell

Pipelines will continue to execute the .bashrc file as if run in an interactive non-login shell but it now behaves as a non-interactive shell. This change may affect scripts that use stdin or have other dependencies on an interactive shell. For these few cases we recommend that you rework your scripts to run non-interactively.

This improves usage of Bitbucket Pipelines in a couple of ways:

• Commands waiting on user input will now exit and fail the build immediately, rather than hanging the build waiting for input.

• Some tools, such as Git and Maven, display download progress indicators in an interactive terminal. Now that builds non-interactively, many tools will no longer log verbose progress indicators, streamlining your Pipelines log output.

Variables with invalid names are no longer passed to the build container

Pipelines started requiring valid C identifiers (matching regex /[A-Za-z_][A-Za-z0-9_]*/) for variable names in November 2016, preventing new invalid variables being created. However, there are still a small number of customers with old, invalid variables configured.

With the recent infrastructure changes, variables with invalid names will no longer be passed to the build container. Scripts that depend on these variables must be updated to use new variables created with valid names.

Public IP addresses

These infrastructure changes mean we can now publish IP addresses for Bitbucket Pipelines. You'll want to know these addresses if you want to provide Pipelines access into your AWS VPC or corporate firewall by adding them to an allowlist, for example.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for the Bitbucket and Pipelines public IP addresses.

Note that our public IP addresses may change in the future.