AI-driven flaky test remediation

AI-driven flaky test remediation allows you to enable our default AI agent or your own agent to fix any flaky tests in your pipeline(s).

• The agent reviews the failing test, its execution history and other related information to hypothesize likely causes for the test being flaky, such as timing, environment or order.

• Once the agent has determined the issue for the flaky test, it will make the necessary changes to fix the test and will raise a pull request.

Use AI-driven flaky test remediation

1. Enable the Flaky test fixer toggle in your Workspace settings, or define the fix-flaky-test trigger in your bitbucket-pipelines.yml file, as shown in the example below.

2. Select Tests on the left sidebar navigation.

3. Select the Actions (…) dropdown and select Fix flaky test.

Example pipeline with the new trigger

If you define the trigger yourself, it takes precedence and Bitbucket does not set anything up, even when the toggle is on.

What Bitbucket sets up for you

When Agentic Pipelines and the Flaky test fixer toggle are both on, and your bitbucket-pipelines.yml does not define a fix-flaky-test trigger, Bitbucket adds configuration equivalent to this:

The condition is true, so the fix is available for any flaky test. This is not written into your repository. It applies when the pipeline is parsed.

To turn it off for a single repository, define the trigger yourself and point it at no pipelines:

To turn it off for the whole workspace, turn the Flaky test fixer toggle off. For the full rules, see Default system agents for Agentic Pipelines.

Using the default agent

We highly recommend using the built-in agent bitbucket-default-fix-flaky-test-agent. The idea is to set it once and allow the Bitbucket Pipelines team to continuously optimize the agent using their CI/CD expertise and all the diagnostic tools they have developed. Although you shouldn’t need to worry too much about the details, but in a nutshell the default fixer agent will:

• Gather more information around the flaky test using built-in tools.

• Execute the test.

• Propose a plan based on analysis.

• Make appropriate changes as per what the agent assessed needs to be done.

• Execute the test to verify the fix.

• Raise a draft pull request and tag the original actor which is whoever initiated the fix for the flaky test.

Change the environment the default agent runs in

The step Bitbucket sets up uses config: default. Define a default step config to change the image, size, and clone behavior the agent runs with. This matters more for flaky tests than for most agents, because the agent runs your test suite to verify the fix, so it needs an image where your tests can run.

See Step config.

Use your own AI agent

You can use your own AI agent as well. To do so, define your own agent under agents definition in their bitbucket-pipelines.yml file.

Example

Tips for using customized (your own) agents

When a pipeline has been triggered via the fix-flaky-test trigger, we will provide you with the following pipelines variables to perform the troubleshooting:

• Test UUID: $BITBUCKET_TRIGGER_TEST_CASE_UUID - The Bitbucket Tests UUID of the flaky test case that triggered this pipeline. You can use this to look up test history, execution details, or call the Bitbucket Tests API.

• Test FQDN (Fully qualified domain name): $BITBUCKET_TRIGGER_TEST_CASE_FQDN - The fully qualified domain name of the flaky test. You can use this to locate the exact test file and method in your codebase that the agent should fix.

• Pull request source branch: $BITBUCKET_TRIGGER_FIX_FLAKY_TEST_SOURCE_BRANCH - The pre-generated branch name the agent must use to push fixes. This is used as the source branch when raising the pull request to ensure linking between the flaky test and the generated pull request.

• Pull request target branch: $BITBUCKET_TRIGGER_FIX_FLAKY_TEST_TARGET_BRANCH - The branch the pull request with the generated code-fix for the flaky test should target. This is typically your default branch where the fix needs to be made.