Bitbucket is getting a new navigation

We’re rolling out these changes, so the documentation may not match your experience in the Bitbucket Cloud app. Read about the new Bitbucket navigation

Quarantining flaky tests

Tests is in open beta and available to Bitbucket Pipelines customers on Standard and Premium plans.

In Bitbucket Pipelines, quarantining isolates flaky tests so you can identify and skip them at runtime, preventing them from blocking the CI. The execution of the quarantined tests can be skipped by the build script. Read about flaky test management.

Mark a test as quarantined

From the Tests page, find the test you want to quarantine in the table or via search. Select More actions, then Mark as quarantined.

Pipelines will identify and skip the tests at runtime, so they don’t block your CI.

Marking a test as quarantined only adds metadata. It doesn’t change how your pipeline runs. You still need to update your build scripts to read this metadata and implement the logic described in the next section.

Enable Quarantine handling in CI

Step 1: Enable test metadata injection in your pipeline

Set inject-classification to true under test-management in the yaml for the pipeline step running the build script. After setting it to true, you can access the list of all quarantined tests at BITBUCKET_TEST_METADATA_FILE_PATH.

pipelines:
  default:
    - step:
        name: Build & Test Without Quarantined Tests
        test-management:
          inject-classification: true

Sample metadata file present at BITBUCKET_TEST_METADATA_FILE_PATH

{
  "quarantinedTests": [
    {
      "package": "com.example",
      "name": "testAdd",
      "class": "CalculatorTest"
    },
    ...
  ]
}

Step 2: Install `jq` in build script

Add the following command to your pipeline step to install jq:

apt-get update && apt-get install -y jq

Step 3: Parse the metadata file

Use jq to extract the list of tests to skip. For example, to extract quarantined tests for Java Maven project:

If you use another test framework or build system, you must parse and fetch quarantined tests accordingly.

The following examples illustrate how to use a test metadata file to skip tests for mvn based projects only. To to similar things in other test frameworks users write advanced scripts to skip specific tests by using class and package names from the metadata file.

QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "!\(.class)#\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "," -)

This command builds a comma-separated list of test patterns to skip, in the format !ClassName#methodName, which is required by Maven.

Quarantined handling in CI for popular frameworks

Junit/TestNG

The -Dtest flag tells Maven to run/skip the specified tests.

mvn clean install -Dtest="$QUARANTINED_TESTS"

Full Bitbucket Pipelines example

Here’s a complete example of a Bitbucket Pipelines step that skips quarantined tests:

pipelines:
  default:
    - step:
        name: Skip quarantined Tests
        image: maven:3.8.4-openjdk-11
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "!\(.class)#\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "," -)
          - mvn clean install -Dtest="$QUARANTINED_TESTS"

PyTest

With -k flag, quarantined tests can be skipped. There are other ways as well like deselect which works with pytest version 6.0+.

pipelines:
  default:
    - step:
        name: Skip Quarantined Tests
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "not \(.class)::\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd " and " -)
          - python3 -m pytest -k $QUARANTINED_TESTS -v

Mocha

Extract test names from the test metadata file and join them into a pipe-separated string, like testA|testB as QUARANTINED_TESTS. Use grep with the invert flag to exclude tests in the package.json file.

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "|" -)
          - npm install
          - npm test -- --grep "$QUARANTINED_TESTS" --invert

TestCafe

Extract test names from the test metadata file and join them into a pipe-separated string, like testA|testB as QUARANTINED_TESTS. Use test-grep flag to exclude tests in the package.json file.

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd " and " -)
          - npm install
          - npm test -- --test-grep '^(?!.*('"$QUARANTINED_TESTS"')).*$'

NUnit

Extract test names from the test metadata file and join them into a &-separated string, like classA+test1&classB+test10 as QUARANTINED_TESTS. Use the --filter flag to exclude quarantined tests. See this reference for skipping tests in the .NET framework using NUnit.

step:
  name: Build and Test
  caches:
    - dotnetcore
  script:
    - apt-get update && apt-get install -y jq
    - QUARANTINED_FILTER=$(jq -r '.quarantinedTests[] | "FullyQualifiedName!=\(.class)+\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "&" -)
    - mkdir -p TestResults test-reports test-results
    - dotnet test --configuration Release --no-build --logger "junit;LogFilePath=TestResults/test-results.xml" --filter "$QUARANTINED_FILTER"

WebdriverIO

Extract test names from the test metadata file and join them into a pipe-separated string, like testA|testB as QUARANTINED_TESTS. Use mochaOpts.grep flag to exclude tests in the package.json file.

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd " and " -)
          - npm install
          - npm test -- --mochaOpts.grep '^(?!.*('"$QUARANTINED_TESTS"')).*$'

Cypress

Cypress does not natively support skipping tests by name via CLI flags. To skip quarantined tests, you can programmatically update your test files to use it.skip() for the specified test names, then revert the changes after the run.

注意:

This approach modifies your test files to skip the specified tests, then restores them after the run.
If your test names contain special regex characters, ensure they are properly escaped.

例:

pipelines:
  default:
    - step:
        name: Skip Quarantined Tests
        caches:
          - node
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          # Extract quarantined test names as a pipe-separated string
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "|" -)
          # Skip quarantined tests by updating test files
          - find cypress/e2e -name "*.cy.js" -exec sed -i '' -E "s/it\('($QUARANTINED_TESTS)'\)/it.skip('\1')/g" {} \;
          # Run Cypress tests with JUnit reporter
          - npx cypress run --reporter mocha-junit-reporter --reporter-options "mochaFile=test-reports/junit/test-results-[hash].xml,testsuitesTitle=true,suiteTitleSeparatedBy=' / '"
          # Revert changes to test files
          - git checkout -- cypress/e2e/

Playwright

Fetch test names from test metadata files and convert them into a pipe-separated string, like testA|testB as QUARANTINED_TESTS. Then use a grep-invert flag to skip tests during the test command.

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "|" -)
          - npm install
          - npm test -- --grep-invert "$QUARANTINED_TESTS"

Jest

Extract test names from the test metadata file and convert them into a pipe-separated string, like testA|testB as QUARANTINED_TESTS. Then, apply a regex pattern like the one below to skip tests during the test command.

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        test-management:
          inject-classification: true
        script:
          - apt-get update && apt-get install -y jq
          - QUARANTINED_TESTS=$(jq -r '.quarantinedTests[] | "\(.name)"' "$BITBUCKET_TEST_METADATA_FILE_PATH" | paste -sd "|" -)
          - npm install
          - npm test -- --testNamePattern='^(?!.*('"$QUARANTINED_TESTS"')).*$'

この内容はお役に立ちましたか?

正確ではなかった明確ではなかった関係なかった

さらにヘルプが必要ですか?

アトラシアンコミュニティをご利用ください。

コミュニティに質問