AI-driven flaky test remediation
Before you begin
AI-driven flaky test remediation is currently in beta.
You must have Agentic Pipelines enabled to use AI-driven flaky test remediation.
AI-driven flaky test remediation allows you to enable our default AI agent or your own agent to fix any flaky tests in your pipeline(s).
The agent reviews the failing test, its execution history and other related information to hypothesize likely causes for the test being flaky, such as timing, environment or order.
Once the agent has determined the issue for the flaky test, it will make the necessary changes to fix the test and will raise a pull request.
Use AI-driven flaky test remediation
Define the
fix-flaky-testtrigger in yourbitbucket-pipelines.ymlfile, as shown in the example below.Select Tests on the left sidebar navigation.
Select the Actions (…) dropdown and select Fix flaky test.
You can also select the Test name and then select the Fix flaky test button in the upper-right corner of the test’s page.
Example pipeline with the new trigger
triggers:
fix-flaky-test:
- condition: "<custom-condition>"
pipelines:
- fix-flaky-test-custom-pipeline
pipelines:
custom:
fix-flaky-test-custom-pipeline:
- step:
auth:
system:
scopes: # scopes required to run the default fixing agent
- read:repository:bitbucket
- write:repository:bitbucket
- read:pipeline:bitbucket
- read:pullrequest:bitbucket
- write:pullrequest:bitbucket
script:
- agent: bitbucket-default-fix-flaky-test-agentUsing the default agent
We highly recommend using the built-in agent bitbucket-default-fix-flaky-test-agent. The idea is to set it once and allow the Bitbucket Pipelines team to continuously optimize the agent using their CI/CD expertise and all the diagnostic tools they have developed. Although you shouldn’t need to worry too much about the details, but in a nutshell the default fixer agent will:
Gather more information around the flaky test using built-in tools.
Execute the test.
Propose a plan based on analysis.
Make appropriate changes as per what the agent assessed needs to be done.
Execute the test to verify the fix.
Raise a draft pull request and tag the original actor which is whoever initiated the fix for the flaky test.
Use your own AI agent
You can use your own AI agent as well. To do so, define your own agent under agents definition in their bitbucket-pipelines.yml file.
Example
definitions:
agents:
custom-flaky-test-fixer-agent:
prompt: |
<Custom Prompt>
triggers:
fix-flaky-test:
- condition: <Custom Condition>
pipelines:
- fix-flaky-test-custom-pipeline
pipelines:
custom:
fix-flaky-test-custom-pipeline:
- step:
auth:
system:
scopes: # scopes required to run the default fixing agent
- read:repository:bitbucket
- write:repository:bitbucket
- read:pipeline:bitbucket
- read:pullrequest:bitbucket
- write:pullrequest:bitbucket
script:
- agent: custom-flaky-test-fixer-agentTips for using customized (your own) agents
When a pipeline has been triggered via the fix-flaky-test trigger, we will provide you with the following pipelines variables to perform the troubleshooting:
Test Uuid:
$BITBUCKET_TRIGGER_TEST_CASE_UUIDTest FQDN (Fully qualified domain name):
$BITBUCKET_TRIGGER_TEST_CASE_FQDNPull request source branch:
$TRIGGER_METADATA_PR_SOURCE_BRANCHPull request target branch:
BITBUCKET_TRIGGER_FIX_FLAKY_TEST_TARGET_BRANCH
The pull request source branch is a special variable that handles linking between the tests and the generated pull request. If you raise the pull request with a different name, the pul request will not be linked to the test.
Was this helpful?