Dynamic pipelines

Dynamic pipelines allow you to leverage Atlassian’s Forge extensibility platform to augment your CI/CD workflows with infinite levels of flexibility and intelligence through making runtime modifications to any component of those workflows using ‘code-level’ application logic.

Without dynamic pipelines, CI/CD configurations are limited to what can be defined in a static .yml syntax. Limited flexibility can be introduced through things like variables and conditions, but the overwhelming majority of the CI/CD workflow is not able to offer any kind of flexibility.

Traditional CI/CD workflow

Dynamic pipelines operate like ‘middleware’ that sits between the static CI/CD configuration files stored in a team’s repositories, and the Bitbucket Pipelines platform that execute their CI/CD builds. By injecting custom logic into that middleware layer, software teams are able to make runtime modifications to their pipeline workflows based on logic they implement into the dynamic pipeline app. The dynamic pipeline is also able to make changes based on external context that the app can retrieve from either Bitbucket Cloud or other external systems. For example, a team might implement a dynamic pipeline that makes a request to their support management tool before any deployments, dynamically injecting a “manual approval” step into the deployment if there are no “on-call” support agents available.

Repository-level dynamic pipelines workflow

Dynamic pipelines aren’t restricted to just the repository level, they can also be configured at the workspace level which means they are available and can be configured across any repositories within that workspace. By leveraging the Bitbucket cloud resource hierarchy, you are able to implement different levels of logic at different levels of your organizations structure, and have those different levels of logic all work seamlessly together to create a true ‘organization-wide’ CI/CD orchestration engine.

Workspace-level dynamic pipelines workflow

Benefits and examples of using dynamic pipelines

Dynamic pipelines is more like a ‘toolkit’ than a ‘single tool’. They are designed to enable teams and organizations to solve entire categories of difficult and complex challenges, rather than a small subset of highly specific use-cases.

Dynamic pipelines enable you to add significant levels of flexibility to traditionally static .yaml pipeline configurations through runtime modification with application-level logic. This capability can be used to solve an enormous range of problems faced either at the level of individual teams or entire organizations.

Some examples of things dynamic pipelines have been used for include:

• Utilizing environment, repository, or workspace variables to define docker image names for use in CI/CD workflows.

• Ensuring all builds run within an organization are executed using docker images from a pre-approved list.

• Ensuring all builds within an organization are executed on self-hosted runners.

• Implementing selective test execution in large monorepos based on pull request changesets.

• Creating standardized and reusable ‘blocks’ of CI/CD workflows that can be defined & shared by platform engineering teams, and then consumed by multiple engineering teams within an organization.

• Dynamically assigning jobs to runner agents tagged with certain cost center codes to automate cloud infrastructure cost attribution.

• Building an entirely customized bitbucket-pipelines.yml syntax that can be used to auto-generate pipeline workflows at runtime from a set of tags and labels.

Creating, using, and configuring dynamic pipelines

Creating a dynamic pipeline app

Dynamic pipeline logic is implemented as code inside an app running in Atlassian’s Forge extensibility platform. Getting up and running with a simple dynamic pipeline app can be achieved in less than thirty minutes. We have an extensive array of resources available to get you started including CLI-generated app templates, step-by-step tutorials, and ready-to-run example repositories that you can clone to your own machine. Head over to the developer documentation to get started.

Using dynamic pipelines

Before configuring and using dynamic pipelines, it’s important to understand how they function so that you can get the desired behavior from them.

The easiest way to think about dynamic pipelines is as a series of tubes connected together, with the bitbucket-pipelines.yml file at one end and the Bitbucket Pipelines platform at the other, similar to what the Workspace-level dynamic pipelines workflow image is showing you above.

Each separate ‘tube’ can take a pipeline workflow configuration in one end, make some changes to that configuration, and then send the updated pipeline configuration out the other end. This ‘pipeline-in/pipeline-out’ design is what makes it possible to connect multiple dynamic pipelines in a row, creating a kind of ‘chain’.

Dynamic pipelines can be configured at the repository level, workspace level or both, and they will always execute ‘bottom-up’ to ensure that modifications defined at higher levels of the hierarchy take precedent over ones made at lower levels.

Dynamic pipeline order of evaluation:

1. bitbucket-pipelines.yml file.

2. Repository-level dynamic pipeline.

3. Workspace-level dynamic pipeline.

Configuring repository-level dynamic pipelines

The simplest implementation of a dynamic pipeline is done at the repository level. Once a dynamic pipeline app has been created following the instructions provided in our developer documentation and is installed into your Bitbucket Cloud workspace, it will be available to configure in your Repository settings. Note: You must be a repository admin to access dynamic pipelines.

1. Select Repository settings, and then select Dynamic Pipelines under Pipelines on the left sidebar navigation.

2. On the Dynamic Pipelines page, select the Repository dynamic pipeline provider dropdown to view the list of all the dynamic pipeline apps installed in your workspace.
   Note: The read-only Workspace dynamic pipeline provider field is displayed on this screen so you can see if there is also a workspace-level dynamic pipeline configured in the workspace. You can use both a repository-level and a workspace-level dynamic pipeline at the same time.

3. Select the dynamic pipeline you want to configure for the repository and then select Save. From that point forward, every pipeline run executed in the context of that repository will use the selected dynamic pipeline app.

Configuring workspace-level dynamic pipelines

Workspace-level dynamic pipelines are an extremely powerful tool for organization-level CI/CD orchestration and management. They can be used to standardize CI/CD across entire companies, ensure compliance and security controls are applied to every build run in a workspace, or even define custom bitbucket-pipelines.yml syntax that is bespoke to your organization.

Important information for Workspace-level dynamic pipelines:

It’s extremely important to understand that a dynamic pipeline configured at the workspace level will run for every single pipeline execution that occurs in that workspace. This makes workspace-level dynamic pipelines an extremely powerful tool, but as we should all know - ‘with great power, comes great responsibility’.

While a poorly implemented dynamic pipeline at the repository level will cause issues for one team or project, a poorly implemented dynamic pipeline at the workspace level can break the builds of an entire organization.

Additionally, it is important to avoid creating circular dependencies where builds/deployments for your dynamic pipeline app itself can be broken due to a bug in that same dynamic pipeline app. For this reason, we advise that you only utilize traditional .yml based pipeline configurations for the repository containing the code for your workspace-level dynamic pipeline app.

Note: You must be an admin of the workspace to configure workspace-level dynamic pipelines.

1. Select Settings in the upper-right corner of the top navigation bar.

2. Select Workspace settings from the Settings dropdown, and then select Dynamic pipelines under Pipelines on the left sidebar navigation.

3. On the Dynamic pipelines page, select the Workspace dynamic pipeline provider dropdown to view the list of all the Dynamic pipeline apps installed in your workspace.

4. Select the dynamic pipeline you want to configure for the workspace and then select Save. From that point forward, every pipeline run executed within that workspace will use the selected dynamic pipeline app.

Working with dynamically generated pipeline configurations

From the perspective of an end user, there is close to zero difference between a pipeline that had its configuration generated the traditional way, purely from .yaml, versus a pipeline that had its configuration generated dynamically.

The main difference is that for a .yaml-based pipeline we provide a link on the pipeline details page to the file in the repository containing the .yaml configuration, incase you ever need to refer to that configuration in order to better understand the pipeline workflow or to debug issues with the pipeline.

With a dynamically generated configuration, the pipeline stored in .yaml is often not a direct representation of what was run by the system as there would have been modifications made to that configuration somewhere between the .yaml file being read, and the pipeline being executed.

To ensure users are still able to properly understand and debug dynamically generated pipelines, we provide a mechanism to allow you to view the final output of the dynamic pipeline generation process so that you can see the exact pipeline configuration that was executed by the system.

1. On the pipeline details page, for a pipeline that had its configuration generated dynamically, you will see a row stating This pipeline is generated by dynamic pipelines. with a View the configuration link below it.

2. Selecting the View the configuration link will open the executed pipeline configuration in a dialog.

Note: The dynamically generated pipeline configuration has been converted back into its .yaml representation for this view (even though we return JSON from the dynamic pipeline app). This is done to make it easier for the developer to read & understand, as the .yaml version of the configuration is what they are used to working with.

3. For extremely long configurations (several thousand lines), the UI will not render the entire configuration. Instead, a Download button will be displayed which will allow you to download the configuration as a file, and then search it from your local text editor or IDE.

Other dynamic pipelines resources

• Step-by-step tutorial

• Dynamic pipelines Forge module reference