Understand and manage flaky tests in Bitbucket Pipelines
Tests is in open beta and available to Bitbucket Pipelines customers on Standard and Premium plans.
A flaky test is one that doesn’t yield the same result every time it is run. Sometimes it passes, and sometimes it fails, even though nothing has changed in the code, the test, or the environment.
This instability typically arises from timing issues, shared state, external dependencies, or environmental quirks, rather than genuine bugs in your application.
Because you can’t trust whether a failure is real, these unstable tests are confusing, noisy, and time‑consuming for developers to investigate.
Why do flaky tests matter?
Over time, flaky tests can hide real bugs and allow broken behavior to reach your users, eroding confidence in your entire test suite. Flaky tests also slow teams down: developers re‑run pipelines to “get a green build,” spend time triaging non‑issues, and may start ignoring failures altogether.
As your CI/CD pipelines and test suites grow, the wasted time and compute add up, increasing costs and delaying releases. Left unchecked, flaky tests can also mask real regressions, which ultimately impacts the quality of the code you ship.
Manage flaky tests
Tests in Bitbucket Pipelines enables you to manually mark and filter tests as flaky, and then take action on them.
The Test summaries view aggregates per‑test data across the latest executions (failure rate, average duration, variance) for up to 250 runs within the last 90 days. Patterns like intermittent failures and high variance make flaky tests stand out without you having to dig through raw logs.
The Test executions view lets you drill into individual runs (build, commit, outcome, duration, last executed) so you can confirm whether a test is truly flaky or failing consistently.
View and mark tests as flaky
To view flaky tests in Bitbucket, go to the Tests page, then filter for Flaky. All tests marked Flaky appear in the Test state column.
To mark a test as flaky:
From the Tests page, find the test.
In the Test state column, select the dropdown.
Select Flaky.
Once you identify and mark a test as flaky, you can fix the test immediately. However, for cases where that’s not possible, you can quarantine the test.
Was this helpful?