Infrastructure changes in Bitbucket Pipelines

Still need help?

The Atlassian Community is here for you.

Ask the community

This page tracks internal infrastructure changes to Bitbucket Pipelines that in rare cases might affect customer builds.

February 2024 - Docker in Docker service upgrade

Starting on Feb 2nd, we will be progressively rolling out an update to the Pipelines Docker in Docker service from 20.10.24 to 25.0.2. For details of the security and bug fixes between these docker versions, visit: Docker Docs — Docker Engine release notes. For common issues related to the upgrade, visit: Recent docker service upgrade in Bitbucket Pipelines

January 2024 - Kubernetes cluster upgrades

Starting on January 3rdwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux for kernel 5.15.0-1050-aws with vulnerability patches for some packages. This change should be unnoticeable to users.

December 2023 - Kubernetes cluster upgrades

Starting on December 5thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Kubernetes version from v1.25.11 to v1.26.10. This change should be unnoticeable to users.

December 2023 - Kubernetes cluster upgrades

Starting on December 5thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1049-aws to 5.15.0-1050-aws. This change should be unnoticeable to users.

November 2023 - Kubernetes cluster upgrades

Starting on November 13thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1047-aws to 5.15.0-1049-aws. This change should be unnoticeable to users.

October 2023 - Kubernetes cluster upgrades

Starting on October 30thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Linux version from v5.15.0-1044-aws to v5.15.0-1047-aws. This change should be unnoticeable to users.

October 2023 - Kubernetes cluster upgrades

Starting on October 18thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Kubernetes version from v1.24.12 to v1.25.11. This change should be unnoticeable to users.

September 2023 - Kubernetes cluster upgrades

Starting on Monday 11th, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1040 to 5.15.0-1043-aws. This change should be unnoticeable to users.

August 2023 - Kubernetes cluster upgrades

Starting on August 8thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1039 to 5.15.0-1040. This change should be unnoticeable to users.

July 2023 - Kubernetes cluster upgrades

Starting on July 25th, we will be upgrading our Kubernetes cluster from v1.23.15 to v1.24.12 for all customers. This change should be transparent to users.

July 2023 - Kubernetes cluster upgrades

Starting on July 17th, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1037 to 5.15.0-1039. This change should be unnoticeable to users.

June 2023 - Kubernetes cluster upgrades

Starting on June 21st, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1036 to 5.15.0-1037. This change should be unnoticeable to users.

June 2023 - Kubernetes cluster upgrades

Starting on June 13th, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1035 to 5.15.0-1036. This change should be unnoticeable to users.

May 2023 - Docker in Docker service upgrade

Starting on May 25th, we will be progressively rolling out an update to the Pipelines Docker in Docker service and an upgraded version of the Docker client for running docker commands from 20.10.23 to 20.10.24. For details of the security and bug fixes between these docker versions, visit: Docker Docs — Docker Engine release notes.

May 2023 - DEPRECATION NOTICE: Bitbucket Cloud Pipelines is ending support for the legacy runtime execution environment in favor of Containerd

Bitbucket Pipelines operates in a Kubernetes-based infrastructure environment, and our internal execution environment leverages Containerd by default for all customers in a transparent manner.

A small number of users have been provisioned onto Docker based runtime infrastructure after a specific exemption was applied by Customer Support; The Docker project has been committed to the Containerd project for a number of years Extending Docker’s Integration with containerd | Docker, we believe most of the tools and libraries should work with Containerd runtime without any issues.  And it’s now time to retire that infrastructure.

This change should be unnoticeable to most of the users and does not impact your ability to build containers using the Docker CLI, the primary impact is if you’re using out-of-date Docker tooling, or tooling that leverages deprecated Docker APIs. 

May 2023 - Kubernetes cluster upgrades

Starting on May 11th, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1033.37 to 5.15.0-1035. This change should be unnoticeable to users.

May 2023 - Kubernetes cluster upgrades

Starting on May 2nd, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1031.35 to 5.15.0-1033.37. This change should be unnoticeable to users.

March 2023 - Kubernetes cluster upgrades

Starting on March 20th, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1030 to 5.15.0-1031. This change should be unnoticeable to users.

March 2023 - Kubernetes cluster upgrades

Starting on March 1st, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1028 to 5.15.0-1030. This change should be unnoticeable to users.

February 2023 - Kubernetes cluster upgrades

Starting on February 27th, we will be upgrading our Kubernetes cluster from v1.22.13 to v.1.23.15 for all customers. This change should be transparent to users.

February 2023 - Docker in Docker service upgrade

Starting on February 20th, we will be progressively rolling out an update to the Pipelines Docker in Docker service and an upgraded version of the Docker client for running docker commands from 20.10.18 to 20.10.23. For details of the security and bug fixes between these docker versions, visit: Docker Docs — Docker Engine release notes.

January 2023 - Kubernetes cluster upgrades

Starting on January 30th, we will be progressively rolling out a container runtime upgrade from containerd-v1.5.13 to containerd-v1.6.11 as well as an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1026 to 5.15.0-1028. These changes should be unnoticeable to users.

December 2022 - Kubernetes cluster upgrades

Starting on December 20th, we will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1023.27 to 5.15.0-1026.30. This change should be unnoticeable to users.

December 2022 - Kubernetes cluster upgrades

Starting on December 07th, we will be progressively rolling out some minor patches to our Kubernetes cluster, kernel version would still be 5.15.0-1023.27. This change should be unnoticeable to users.

November 2022 - Kubernetes cluster upgrades

Starting on November 29th, we will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Ubuntu AMI’s Linux kernel version from 5.15.0-1022.26 to 5.15.0-1023.27. This change should be unnoticeable to users.

November 2022 - Kubernetes cluster upgrades

Starting on November 9th, we will be progressively rolling out a security upgrade to the network fabric of our Kubernetes cluster. This change should be transparent to users.

October 2022 - Kubernetes cluster upgrades

Starting on October 31st, we will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Ubuntu AMI’s Linux kernel version from 5.15.0-1020 to 5.15.0-1022. This change should be unnoticeable to users.

October 2022 - Kubernetes cluster upgrades

Starting on Oct 6th, we will be upgrading our Kubernetes cluster AMI kernel from 5.15.0-1019.23 to 5.15.0-1020.24 for all customers. This change should be transparent to users.

September 2022 - Kubernetes cluster upgrades

Starting on September 26th, we will be upgrading our Kubernetes cluster from v1.21.8 to v1.22.13 for all customers. This change should be transparent to users.

September 2022 - Docker in Docker service upgrade

Starting on September 23rd, we will be progressively rolling out an update to the Pipelines Docker in Docker service and an upgraded version of the Docker client for running docker commands from 20.10.15 to 20.10.18. For details of the security and bug fixes between these docker versions, visit: Docker Docs — Docker Engine release notes.

September 2022 - Kubernetes cluster upgrades

Starting on September 14thwe will be progressively rolling out an upgrade to our Kubernetes cluster to upgrade the Ubuntu AMI's Linux kernel version from 5.15.0-1017.21 to 5.15.0-1019.23. This change should be unnoticeable to users.

August 2022 - Kubernetes cluster upgrades

Starting on August 31stwe will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Ubuntu AMI’s Linux kernel version from 5.15.0-1015.19 to 5.15.0-1017.21. This change should be unnoticeable to users.

August 2022 - Kubernetes cluster upgrades

Starting on August 1stwe will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Ubuntu AMI’s Linux kernel version from 5.13.0.1031.35 to 5.15.0-1015.19. This change should be unnoticeable to users.

July 2022 - containerd version upgrade

Starting on July 26th, we will be upgrading containerd on the Kubernetes nodes from v1.5.11 to v1.5.13. This change should be unnoticeable to users.

July 2022 - Kubernetes cluster upgrades

Starting on 4th July, we will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Ubuntu AMI’s Linux kernel version from 5.13.0.1022.24 to 5.13.0.1031.35. This change should be unnoticeable to users.

June 2022 - Git client upgrade

Starting on June 29th, we will be progressively rolling out an upgrade to Git (2.15.4 to 2.36.1) and Git LFS (2.13.2 to 3.1.4) used when cloning the repository during build setup. For runners users, during the rollout, they may notice the tag of the agent-setup image from prod-stable to stg-stable and switch back to prod-stable at the completion of the rollout. This upgrade does not require any changes from customers except Git LFS users. Users of Git LFS may experience pipeline failures. For information on this issue, visit: 'pre-push hook already exists' error.

June 2022 - Docker in Docker service upgrade

Starting on June 20th, we will be progressively rolling out BuildKit for the Pipelines Docker in Docker service. For details on BuildKit, visit: Docker Docs — Build images with BuildKit.

June 2022 - Docker client upgrade

Starting on June 13th, we will be incrementally rolling out an upgraded version of the Docker client available for running docker commands from v19.03.15 to v20.10.15. For more information about this upgrade, see the Bitbucket Pipelines: Docker client upgrade post.

May 2022 - containerd version upgrade

Starting on May 31st, we will be upgrading containerd on the Kubernetes nodes from v1.5.10 to v1.5.11. This change should be unnoticeable to users.

May 2022 - Kubernetes cluster upgrade

Starting on May 24th, we will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Linux kernel to a newer patch version (5.13.0.1022). This change should be unnoticeable to users.

May 2022 - Docker in Docker service upgrade

Starting on May 23rd, we will be progressively rolling out an update to the Pipelines Docker in Docker service from version 20.10.6 to 20.10.15. For details of the security and bug fixes between these docker versions, visit: Docker Docs — Docker Engine release notes.

April 2022 - containerd version upgrade

Starting on April 4th, we will be upgrading containerd on the Kubernetes nodes from v1.5.8 to v1.5.10. This change should be transparent to users.

March 2022 - Kubernetes cluster upgrade

Starting on March 28th, we will be progressively rolling out an upgrade to our Kubernetes cluster which upgrades the Linux kernel from version 5.11 to 5.13. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

February 2022 - Docker in Docker service upgrade

Starting on February 22, we will be progressively rolling out an update to the Pipelines Docker in Docker service from version 20.10.5 to 20.10.6. This contains a fix that will validate Dockerfiles containing Buildkit directives which are not yet supported in Bitbucket Pipelines, causing pipelines to fail. If your pipeline contains any Buildkit directives, remove them and run the pipeline again.

February 2022 - Kubernetes cluster upgrade

Starting on February 15, we will be progressively upgrading our Kubernetes cluster from v1.20.11 to v1.21.8 for all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

December 2021 - Kubernetes cluster upgrades

Starting 13th December, will be upgrading our Kubernetes infrastructure operating system from Flatcar to Ubuntu. This involves a kernel upgrade from 5.4.92 to 5.11.0, and will be rolled out progressively. We expect this change to be transparent for users.

December 2021 - New outbound IP Addresses

On 8th December 2021, new IP addresses were provisioned for our build infrastructure, allowing for improved resiliency in our Kubernetes infrastructure. These addresses will become active in the next six weeks.

Refer to What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for the updated listing of outbound IPs.

November 2021 - containerd patch version upgrade

Starting on November 8, we will be upgrading containerd on the Kubernetes nodes from v1.5.0 to v1.5.7. This change should be transparent to users.

August 2021 - Kubernetes cluster upgrade

Starting on August 30, we will be upgrading our Kubernetes cluster from v1.19.7 to v1.20 for all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

August 2021 - Container runtimes upgrade

Starting on August 24th, we will be rolling out an upgraded version of runc1.0.0-rc10 to runc1.0.0-rc95 and containerd1.4.3 to containerd1.5.0. For more information about this upgrade, see the Bitbucket Pipelines: containerd and runc upgrades community post.

May 2021 - Flatcar OS upgrade rollback

After the upgrade to Kubernetes FlatcarOS v2765.2.2, we detected errors with Ruby/Rails failing with ActiveStorage::IntegrityError and yarn builds hanging. Because of this, we are rolling back to Kubernetes FlatcarOS v2605.12.0.

May 2021 - Flatcar OS upgrade

Starting on May 4th, we will be upgrading our Kubernetes FlatcarOS to v2765.2.2 for all customers. This change should be transparent to users.

March 2021 - containerd minor version upgrade

Starting on March 29th, we will be upgrading containerd on the Kubernetes nodes from v1.3.9 to v1.4.3. Within this latest version upgrade, containerd ended support for certain versions of some old images, like python:3.5.1 and redis:2.8.23. Update to the latest version to fix any issues.

March 2021 - Kubernetes cluster upgrade

Starting on March 1st, we will be upgrading our Kubernetes cluster from v1.18.0 to v1.19.7 for all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

February 2021 - Runtime environment migration to Kata Container

Starting on March 1st, we will be rolling out the migration from Docker & Containerd to Kata Container lightweight virtual machines for increased isolation and security of customer workloads to our Standard and Premium Bitbucket Cloud customers. Any users on a Free plan have already been migrated. This change should be transparent to users.

February 2021 - Flatcar OS upgrade

Starting on February 17th, we will be upgrading our Kubernetes FlatcarOS to v2605.12.0 for all customers. This change should be transparent to users.

February 2021 - Git LFS client upgrade

As of February 2021, we upgraded the Git LFS client from version 2.6.0 to 2.13.2. This change should be transparent to users.

February 2021 - Changes in docker’s daemon

Starting on February 3rd, we will be incrementally rolling out an upgraded version from 19.03.13 to 19.03.15.

January 2021- FlatcarOS upgrade

Starting on January 27th, we will be upgrading our Kubernetes FlatcarOS to v2605.10.0 for all customers. This change should be transparent to users.

January 2021 - containerd minor version upgrade

Starting on January 20th, we will be upgrading containerd on the Kubernetes nodes from v1.3.4 to v1.3.9.

January 2021 - Kubernetes cluster upgrade

Starting on January 20th, we will be upgrading our Kubernetes cluster from v1.17.9 to v1.18.10 for all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

December 2020 - Kubernetes cluster added

We’ve added a new Kubernetes cluster to run builds across some pipelines. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

November 2020 - Kubernetes node OS upgrade

Starting on November 11th, we will be rolling out an upgraded version of the operating system that our Kubernetes build nodes run. This involves a change of Linux kernel versions from 4.19.143 to 5.4.72. This change should be transparent to users.

November 2020 - Changes in docker’s daemon

Starting on November 2nd through November 10th, we will be incrementally rolling out an upgraded version from 18.09.9 to 19.03.13.

September 2020 - Changes in docker’s daemon

The docker daemon version used in the docker service has been upgraded from 18.09.1 to the latest 18.09.9

September 2020 - changes to docker in docker container filesystem permissions

Starting on 2nd September the docker in docker container's root filesystem will be read-only except for the directories required to use docker in order to harden security. The BITBUCKET_CLONE_DIR and its subdirectories, any other volumes you create, will remain writable.

April 2020 - containerd minor version upgrade

Starting on 22rd April containerd on the kubernetes nodes was upgraded from 1.2.x to 1.3.4.

Starting on 29th April switch to containerd-shim-runc-v2 to use per-pod shim instead of per-container shim.

November 2019  - Kubernetes cluster upgrades

Starting on 20th November, we will be progressively rolling out Kubernetes cluster upgrades to all customers. This change should be transparent to users. If you are running behind a corporate firewall and haven’t recently allowlisted pipelines IP addresses, you can review these at What are the IP addresses to configure a corporate firewall?

August 2019  - containerd runtime changes on nodes

Starting from August 7th, the nodes in our kubernetes build cluster will be changed to run with containerd rather than docker. The change should be transparent to most Bitbucket Pipelines users.
Update August 9th: We've identified 2 issues with the rollout:

  • Users with docker images hosted on outdated Sonatype Nexus instances will have trouble pulling images via  containerd. Please file a support ticket at https://support.atlassian.com/contact to be excluded from the migration and consider upgrading the Nexus instance (see https://issues.sonatype.org/browse/NEXUS-12684).

  • Users with private docker images hosted on bintray.com will experience 401 Unauthorized response with containerd. We've identified the issue to be with the provider and are communicating with them on resolution. Please file a support ticket to be excluded from the containerd until the issue is resolved using the following link https://support.atlassian.com/contact.

April 2019 - SSH keyscan performed from build environment

Starting from the 17th of April 2019, SSH keyscans will be performed from within the build environment. This means you will need to add the valid IP addresses to an allowlist for Bitbucket Pipelines build environments to continue using this. The IP addresses in use by Bitbucket Pipelines services will no longer need to be allowlisted.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

October 2018 - New outbound IP addresses

Starting from the 8th of November 2018, new IP addresses will be in use by Bitbucket Pipelines services (not our build infrastructure).

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

September 2018 - Docker in Docker User Namespace Remapping

On 27th September 2018, we rolled out a change to enable user namespace remapping in our docker in docker daemon we provide to users as part of a steps execution to harden the security of pipelines.

September 2018 - New outbound IP addresses

On 12th September 2018, new IP addresses were provisioned for our build infrastructure to offer future multi-region failover in our Kubernetes infrastructure. These addresses will become active in the next two weeks.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

August 2018 - Performance improvements by moving to EC2 M5d instance types

On 1st August 2018, we swapped our Kubernetes nodes from using EC2 M4 instance types to M5d's. M5d instances use NVMe drives (instead of EBS volumes), which are much faster, as well as located on the underlying compute hardware, not having then the overhead of transferring data on the drives over a storage network.

March 2018 - New outbound IP addresses

On 15th March 2018, new IP addresses were provisioned for our build infrastructure. These addresses will become active in the next two weeks.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

November 2017 - Docker now a service, limited to 1 GB memory

On 28 November 2017, as part of implementing docker-run support in Pipelines, we now treat Docker as a Pipelines service. This means commands executed via Docker will have a memory limit of 1 GB, and builds that enable Docker can only use two additional services per build step.

There are a very small number of existing builds that use three services and have Docker enabled that will break with this change. We have directly notified customers who have recently run builds with this configuration.

Our recommendation is to either stop running one of your services or change one service to run using "docker run" instead (see Run Docker commands in Bitbucket Pipelines). Docker run support will also give you the flexibility to start multiple Docker containers in the same build, including via docker-compose files.

October 2017 - New outbound IP addresses

On 25 October 2017, new IP addresses were provisioned for our build infrastructure.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for details.

September 2017 - Docker upgrade

On 7 September 2017, we upgraded the Docker daemon provided to Pipelines build containers, from 1.12.6 to 17.05.

Please see this ticket for more details: https://bitbucket.org/site/master/issues/14333/upgrade-docker-for-multi-stage-builds

February 2017 - New infrastructure

As of February 2017, we're rolling out changes to Pipeline's build infrastructure to provide a foundation for upcoming new features. Pipelines still executes your scripts in an isolated Docker container, and most people won't notice any change in behavior.

There are a couple of minor differences that may affect some people, described below.

How to tell if you have the new infrastructure

You can tell if you've got the updated infrastructure by looking at the log file. The 'Build setup' section at the top will be noticeably shorter, and will no longer contain docker run commands.

You still have the old infrastructure if you see docker run commands in the 'Build setup' section of your log file similar to the following:

What's changed?

Scripts are no longer run in an interactive shell

Pipelines will continue to execute the .bashrc file as if run in an interactive non-login shell but it now behaves as a non-interactive shell. This change may affect scripts that use stdin or have other dependencies on an interactive shell. For these few cases we recommend that you rework your scripts to run non-interactively.

This improves usage of Bitbucket Pipelines in a couple of ways:

  • Commands waiting on user input will now exit and fail the build immediately, rather than hanging the build waiting for input.

  • Some tools, such as Git and Maven, display download progress indicators in an interactive terminal. Now that builds non-interactively, many tools will no longer log verbose progress indicators, streamlining your Pipelines log output.

Variables with invalid names are no longer passed to the build container

Pipelines started requiring valid C identifiers (matching regex /[A-Za-z_][A-Za-z0-9_]*/) for variable names in November 2016, preventing new invalid variables being created. However, there are still a small number of customers with old, invalid variables configured.

With the recent infrastructure changes, variables with invalid names will no longer be passed to the build container. Scripts that depend on these variables must be updated to use new variables created with valid names.

Public IP addresses

These infrastructure changes mean we can now publish IP addresses for Bitbucket Pipelines. You'll want to know these addresses if you want to provide Pipelines access into your AWS VPC or corporate firewall by adding them to an allowlist, for example.

See What are the Bitbucket Cloud IP addresses I should use to configure my corporate firewall? for the Bitbucket and Pipelines public IP addresses.

Note that our public IP addresses may change in the future.

Last modified on Feb 9, 2024

Was this helpful?

Yes
No
Provide feedback about this article
Powered by Confluence and Scroll Viewport.