Bitbucket is getting a new navigation

We’re rolling out these changes, so the documentation may not match your experience in the Bitbucket Cloud app. Read about the new Bitbucket navigation

Get started with tests in Pipelines

Tests is in open beta and available to Bitbucket Pipelines customers on Standard and Premium plans.

How test reporting works in a Pipeline step

ビルドステップでテスト結果を生成すると、Pipelines はそれを自動的に検知し、Web インターフェイスで失敗状況を表示します。現在、これは記述されている言語に関係なく、JUnit と Maven Surefire XML フォーマットで動作します。

リポジトリの読み取りアクセス権を持つすべてのユーザーが、パイプラインのログパネルでテスト結果を確認できます。テストの失敗結果が見つかった場合、ログビューには失敗したテストがハイライトで示され、失敗理由と、存在する場合はスタックトレースが表示されます。[ビルド] タブから完全なログビューを参照することもできます。

[テスト] タブは、レビューが必要な、失敗したテストがある場合にのみ表示されます。

失敗したテストの制限

最初の 100 件の失敗したテスト結果のみが表示されます。

Perquisites

Your test framework must generate JUnit-style XML reports.
Bitbucket Pipelines must be enabled for your repository.

Go to your Bitbucket repository and select Test management.
Review the Test summaries page to see aggregated data for each test, including failure rate, average duration, and variance duration.
Use the search bar or filters to find specific tests by name, label, or state.
Select a test summary to view detailed test executions for that test, including buildNumber, commit message, lastExecuted, execution duration, and outcome.
[Optional] To mark a test as Flaky, go to the Test state column, then choose Flaky from the dropdown menu.
[Optional] To mark a test as Quarantined, select More actions (…) for the test and mark it as Quarantined.

If the repository meets the eligibility criteria and the frameworks generate XML format reports as described below, then the tests will be generated automatically.

Configure test reporting

テストレポートを有効化するには、ビルドテストのレポートが、次のいずれかのサポート対象の既定ロケーション (ディレクトリの階層は 3 レベル) に生成されることを確認します。

./**/surefire-reports/**/*.xml

./**/failsafe-reports/**/*.xml

./**/test-results/**/*.xml

./**/test-reports/**/*.xml

./**/TestResults/**/*.xml

テストレポートのファイルスキャナは、ビルドのベースディレクトリ (/opt/atlassian/pipelines/agent/build) から検索を開始します。

If the test reports are in some path other than the default path mentioned above, it can be defined like the example provided below:

例

artifacts:
  upload:
   - name: "custom test reports"
     type: "test-reports"
     paths:
     - "custom_path/*.xml"
     - "**/custom_path2/*.xml"

Configuring test runners to generate XML format test reports

Some test frameworks, such as TestNG, generate multiple JUnit XML reports containing the same tests but with different suite names.

In this case, you will see multiple tests with duplicated test names but different suites on the test management page, which can cause confusion.

Maven Surefire プラグイン

For Maven build jobs, no special configuration is required if you are using Maven Surefire Plugin. The reports are automatically generated when the unit or integration maven test goals are executed.

PHPUnit

For PHPUnit test reports, you should explicitly specify the --log-junit parameter to generate the test reports output to a particular location.

bitbucket-pipelines.yml の例

image: php:7.1.1
pipelines:
  default:
    - step:
        script:
          - apt-get update && apt-get install -y unzip
          - curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
          - composer require phpunit/phpunit
          - vendor/bin/phpunit --log-junit ./test-reports/junit.xml
    

dotnet

dotnet テストレポートでは、JUnitTestLogger を使用して互換性のある XML テスト結果を生成します。

JUnitTestLogger を使用して JUnit フォーマットを直接出力する:

bitbucket-pipelines.yml の例
image: mcr.microsoft.com/dotnet/sdk pipelines: default: - step: script: - dotnet add package JUnitTestLogger --version 1.1.0 - dotnet test --logger "junit"

Junit/TestNG (with Maven)

By default, JUnit and TestNG generate test reports in XML format. Run the command below to generate test reports in target/surefire-reports.

mvn clean test

Below is an example of how you can add it to your Bitbucket Pipelines script:

pipelines:
  default:
    - step:
        name: Build and Test
        image: maven:3.8.4-openjdk-11
        script:
          - mvn clean install 

PyTest

Generate a JUnit XML report with Pytest using the --junit-xml flag. Specify the path to store the test report, for example, test-results/<report-name>.xml.

Below is an example of how you can add it to your Bitbucket Pipelines script:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - pip
        script:
          - pytest --junit-xml=test-results/junit.xml 

Mocha

Add the following package in dependencies in package.json: “mocha-junit-reporter": "^2.2.1" or use npm to install it using npm install --save-dev mocha-junit-reporter.

...
  "scripts": {
    "test": "mocha test/**/*.test.js --reporter mocha-junit-reporter,
  },
  "devDependencies": {
    "mocha-junit-reporter": "^2.2.1"
  },
...

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

TestCafe

Install or include the following dependency: npm install --save-dev testcafe-reporter-junit
Then update package.json with the following test script:

{
  "scripts": {
    "test": "testcafe chrome:headless tests --reporter junit:test-reports/junit.xml",
  }
}

The above applies to Google Chrome. Configuration may vary slightly by browser. Reports will be created in the test-reports folder or the specified path in the command.

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

Jest

Use jest-junit to generate a report in Junit XML format.

...
  "scripts": {
    "test": "jest --reporters=default --reporters=jest-junit",
  },
  "devDependencies": {
    "jest": "^29.7.0",
    "jest-junit": "^16.0.0"
  },
...

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

WebdriverIO

Install @wdio/junit-reporter as a devDependency in your package.json using the command below:

npm install @wdio/junit-reporter --save-dev

2. Configure the WebdriverIO config file:

export const config = {
  // ... other config options
  
  reporters: [
    'spec', // Optional: for console output
    ['junit', {
      outputDir: 'test-results',        // Directory where reports will be saved
      outputFileFormat: () => 'junit.xml' // Filename format
    }]
  ],
  
  // ... rest of your config
};

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

Playwright

The following line needs to be added in Playwright config under reporter section in the playwright.config.ts file to generate this test report:

['junit', { outputFile: 'test-results/junit.xml' }]

Run tests using npm test in your pipeline script.

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npm test

Cypress

Add the following package in dependencies in package.json: “mocha-junit-reporter": "^2.2.1"
or use npm to install it using npm install --save-dev mocha-junit-reporter.
Run Cypress with the required options in your Bitbucket Pipelines configuration:

npx cypress run --reporter mocha-junit-reporter --reporter-options \
  "mochaFile=test-reports/junit/test-results-[hash].xml,\
   testsuitesTitle=true,\
   suiteTitleSeparatedBy=' / ',\
   rootSuiteTitle='Cypress Tests'"

Example of bitbucket-pipelines.yaml:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - node
        script:
          - npm install
          - npx cypress run --reporter mocha-junit-reporter --reporter-options "mochaFile=test-reports/junit/test-results-[hash].xml, testsuitesTitle=true, suiteTitleSeparatedBy=' / ', rootSuiteTitle='Cypress Tests'"

NUnit

Use the popular JunitXml.TestLogger (71M downloads) from NuGet Package Manager to generate test reports in JUnit XML format. The latest stable version is 7.0.2. More details are available here.

// Install the Junit Test Logger via package reference:
<PackageReference Include="JunitXml.TestLogger" Version="7.0.2" />

Update your pipeline script to generate the report in JUnit XML format. See the example below:

pipelines:
  default:
    - step:
        name: Build and Test
        caches:
          - dotnetcore
        script:
          - echo "Restoring NuGet packages..."
          - dotnet restore
          - echo "Building solution..."
          - dotnet build --configuration Release --no-restore
          - echo "Running tests with JUnit XML output..."
          - mkdir -p TestResults test-reports test-results
          - dotnet test --configuration Release --no-build --logger "junit;LogFilePath=TestResults/test-results.xml"

Selenium/Appium

Selenium is a tool that simulates browser actions. It, however, is not a testing framework; rather, it must be used in combination with testing frameworks like JUnit, TestNG, Mocha, or pytest to run organized tests.

These testing frameworks have their own setting for generating JUnit XML style reports, however, Selenium in itself can’t generate any reports as it is nothing more than a tool that simulates actions.

Similarly, Appium supports mobile testing with test frameworks such as WebdriverIO and Mocha. Ensure the test framework generates Appium test reports in JUnit XML format.

その他のテストランナー

その他のテストランナーの場合は、互換性を持つ XML 形式のテスト結果が、既定のサポート対象のロケーションのいずれかにビルドの一部として生成されることを確認します。

この内容はお役に立ちましたか?

正確ではなかった明確ではなかった関係なかった

さらにヘルプが必要ですか?

アトラシアンコミュニティをご利用ください。

コミュニティに質問

Bitbucket is getting a new navigation

Get started with tests in Pipelines

How test reporting works in a Pipeline step

Configure test reporting

Configuring test runners to generate XML format test reports

Maven Surefire プラグイン

PHPUnit

dotnet

Junit/TestNG (with Maven)

PyTest

Mocha

TestCafe

Jest

WebdriverIO

Playwright

Cypress

NUnit

Selenium/Appium

その他のテスト ランナー

さらにヘルプが必要ですか?

その他のテストランナー