Configure logging in self-hosted runners

The functionality outlined here is NOT SUPPORTED by the Bitbucket Cloud support team.

The functionalities documented on this page are for customers with highly sophisticated on-premise runner operations who are comfortable with extremely high levels of technical complexity.

This documentation is provided as a reference only.

Exposing build logs locally on the runner

Usually, build logs from Pipelines executions done on the Pipelines runner are sent to Bitbucket Cloud and can be retrieved via the Bitbucket Cloud API.

However, there may be situations where you want to collect those build logs locally from the machine running the Pipelines runner itself, without having to call the Bitbucket Cloud API.

To do this, you can configure the runner to capture build logs as part of the overall runner system logs, and then ingest them using your log management tool of choice.

Linux Docker runner

Add -e BITBUCKET_PIPELINES_RUNNER_BUILD_LOG_ENABLED=TRUE to the usual runner docker run command

Example

docker container run \
  ... 
  -e BITBUCKET_PIPELINES_RUNNER_BUILD_LOG_ENABLED=TRUE \
  ...
  runner:latest

Linux bash runner / macOS bash runner / Windows PowerShell runner

Set BITBUCKET_PIPELINES_RUNNER_BUILD_LOG_ENABLED environment variable to TRUE, before running the usual runner start.sh script

Example

export BITBUCKET_PIPELINES_RUNNER_BUILD_LOG_ENABLED=TRUE
start.sh

# or

BITBUCKET_PIPELINES_RUNNER_BUILD_LOG_ENABLED=TRUE start.sh

Configuring the logging output and format

The Pipelines runner supports configuration of logging through the use of a customer provided logger configuration file.

Logback (https://logback.qos.ch) is the logging utility used for logging in the Pipelines runner. To customize your runner logging configuration, you need to do the following:

Prepare an .xml configuration file accessible from the runner. See the corresponding steps below.
Configure the LOGBACK_CONFIGURATION_FILE environment variable for the runner to use that file. See the corresponding steps below.

1. Prepare the Configuration File

A local .xml is need. (e.g my-logback-config.xml).
Atlassian have provided a set of pre-made configuration files, including the default configuration, at the bottom of this section.
See Chapter 3: Configuration for syntax guidelines.

2. Use the Configuration File

Linux Docker runner

Mount the logback config file so that it can be accessed from the runner and pass an environment variable LOGBACK_CONFIGURATION_FILE argument that points to the path of the configuration file to the docker run command.

Example

docker container run \
  ... 
  -v /path/on/host/to/custom-logback.xml:/tmp/custom-logback.xml:ro \
  -e LOGBACK_CONFIGURATION_FILE=/tmp/custom-logback.xml \
  ...
  runner:latest

Linux bash runner / macOS bash runner / Windows PowerShell runner

Set LOGBACK_CONFIGURATION_FILE environment variable to /path/on/host/to/custom-logback.xml

Example

export LOGBACK_CONFIGURATION_FILE=/path/on/host/to/custom-logback.xml
start.sh

# or

LOGBACK_CONFIGURATION_FILE=/path/on/host/to/custom-logback.xml start.sh

Example configuration files

Example of a default logback config file

This is a copy of the default Logback configuration file used in the runner without any changes.

<configuration>
    <define name="BASE_LOG_PATH" class="com.atlassian.pipelines.runner.core.log.BaseLogPathPropertyDefiner"/>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
            <evaluator>
                <expression>com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException.class.isInstance(throwable)</expression>
            </evaluator>
            <onMatch>DENY</onMatch>
        </filter>
        <encoder>
            <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
            <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
        </encoder>
    </appender>
    <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${BASE_LOG_PATH}/runner.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${BASE_LOG_PATH}/runner.%d.%i.log</fileNamePattern>
            <maxHistory>7</maxHistory>
            <maxFileSize>100MB</maxFileSize>
            <totalSizeCap>1GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
            <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
        </encoder>
    </appender>
    <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
        <resetJUL>true</resetJUL>
    </contextListener>
    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
    </root>
    <logger name="com.atlassian.pipelines.runner" level="${BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL:-INFO}" />
    <logger name="build" level="INFO" />
    <logger name="build.service.auth-proxy" level="OFF" />
    <logger name="com.netflix" level="OFF" />
    <logger name="com.atlassian.pipelines.stargate" level="OFF" />
    <logger name="com.github.dockerjava" level="OFF" />
</configuration>

Example of a verbose logging config file

This is a copy of a Logback configuration file setup for verbose logging.

<configuration>
  <define name="BASE_LOG_PATH" class="com.atlassian.pipelines.runner.core.log.BaseLogPathPropertyDefiner"/>
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator>
        <expression>com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException.class.isInstance(throwable)</expression>
      </evaluator>
      <onMatch>DENY</onMatch>
    </filter>
    <encoder>
      <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
      <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
    </encoder>
  </appender>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${BASE_LOG_PATH}/runner.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${BASE_LOG_PATH}/runner.%d.%i.log</fileNamePattern>
      <maxHistory>7</maxHistory>
      <maxFileSize>100MB</maxFileSize>
      <totalSizeCap>1GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
      <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
    </encoder>
  </appender>
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <resetJUL>true</resetJUL>
  </contextListener>
  <root level="INFO">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="FILE" />
  </root>
  <logger name="com.atlassian.pipelines.runner" level="${BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL:-INFO}" />
  <logger name="build" level="INFO" />
</configuration>

Example of a JSON logging config file

Output logs as JSON to stdout instead of plain text.

<configuration>
  <define name="BASE_LOG_PATH" class="com.atlassian.pipelines.runner.core.log.BaseLogPathPropertyDefiner"/>
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator>
        <expression>com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException.class.isInstance(throwable)</expression>
      </evaluator>
      <onMatch>DENY</onMatch>
    </filter>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
  </appender>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${BASE_LOG_PATH}/runner.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${BASE_LOG_PATH}/runner.%d.%i.log</fileNamePattern>
      <maxHistory>7</maxHistory>
      <maxFileSize>100MB</maxFileSize>
      <totalSizeCap>1GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
      <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
    </encoder>
  </appender>
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <resetJUL>true</resetJUL>
  </contextListener>
  <root level="INFO">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="FILE" />
  </root>
  <logger name="com.atlassian.pipelines.runner" level="${BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL:-INFO}" />
  <logger name="build" level="INFO" />
  <logger name="build.service.auth-proxy" level="OFF" />
  <logger name="com.netflix" level="OFF" />
  <logger name="com.atlassian.pipelines.stargate" level="OFF" />
  <logger name="com.github.dockerjava" level="OFF" />
</configuration>

Example of a configured log path config file

Change base log path to /tmp

<configuration>
  <variable name="BASE_LOG_PATH" value="/tmp" />
  <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <filter class="ch.qos.logback.core.filter.EvaluatorFilter">
      <evaluator>
        <expression>com.atlassian.pipelines.stargate.client.core.exceptions.StargateForbiddenException.class.isInstance(throwable)</expression>
      </evaluator>
      <onMatch>DENY</onMatch>
    </filter>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />
  </appender>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${BASE_LOG_PATH}/runner.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${BASE_LOG_PATH}/runner.%d.%i.log</fileNamePattern>
      <maxHistory>7</maxHistory>
      <maxFileSize>100MB</maxFileSize>
      <totalSizeCap>1GB</totalSizeCap>
    </rollingPolicy>
    <encoder>
      <!-- print [buildLogName] from SystemLogAppenderFilter if exists, otherwise do not print empty [] -->
      <pattern>[%date{ISO8601}]%replace([%X{buildLogName}]){'\[\]',''} %message%n%rootException</pattern>
    </encoder>
  </appender>
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <resetJUL>true</resetJUL>
  </contextListener>
  <root level="INFO">
    <appender-ref ref="STDOUT" />
    <appender-ref ref="FILE" />
  </root>
  <logger name="com.atlassian.pipelines.runner" level="${BITBUCKET_PIPELINES_RUNNER_LOG_LEVEL:-INFO}" />
  <logger name="build" level="INFO" />
  <logger name="build.service.auth-proxy" level="OFF" />
  <logger name="com.netflix" level="OFF" />
  <logger name="com.atlassian.pipelines.stargate" level="OFF" />
  <logger name="com.github.dockerjava" level="OFF" />
</configuration>

Was this helpful?

It wasn't accurate

It wasn't clear

It wasn't relevant

Still need help?

The Atlassian Community is here for you.

Ask the Community