Troubleshooting runners

Checking the status of a runner

The runner status can be checked on the Runners page associated with the workspace or repository.

  • To find the status of a workspace runner, select Settings on the left sidebar of the workspace, and select Workspace runners under Pipelines on the left sidebar navigation.

  • To find the status of a repository runner, select Repository settings on the left sidebar of the repository, and select Runners under Pipelines on the left sidebar navigation.

The status can be one of the following:

  • Unregistered: The runner was created but was never run. Note: If a runner isn’t registered within one week, it will be removed.

  • Online: The runner is live and available for step scheduling.

  • Offline: The runner is not available. It may have been stopped, or there may be network connectivity issues.

  • Disabled: The runner was temporarily disabled, and steps will not be scheduled on it until it is enabled again.

runner display in the user interface with online status

Reviewing runner logs

Build and services logs (not including the runners' logs) for a step are removed from a runner host once the step is complete. These logs can be found in the step logs on the Pipeline's user interface.

The location of the runner log files varies between operating systems and the type of runner.

Reviewing log files for Linux Docker runners

  1. Access the Linux device hosting the affected runner.

  2. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    1 <runner_working_directory>/<runnerUuid>/runner.log

    For example:

    1 /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

    The default working directory for the runner is the /tmp directory.

  3. Review the end of the log file for errors that need to be addressed.

Reviewing log files for Windows PowerShell runners

  1. Access the Windows device hosting the affected runner.

  2. Using PowerShell or File Explorer, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where PowerShell was working when the runner was started.
    For example: if PowerShell was working in the users' home directory, such as:

    1 PS C:\Users\Username>

    The runner installation directory would be C:\Users\Username\atlassian-bitbucket-pipelines-runner\

  3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    1 <runner_working_directory>\<runnerUuid>\runner.log

    For example:

    1 .\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log

    The default working directory for the runner is the \temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

  4. Review the end of the log file for errors that need to be addressed.

Reviewing log files for macOS shell runners

  1. Access the macOS device hosting the affected runner.

  2. Using Terminal or Finder, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where Terminal was working when the runner was started.
    For example: if Terminal was working in the users' home directory (~/), such as: /Users/Username/; the runner installation directory would be /Users/Username/atlassian-bitbucket-pipelines-runner/

  3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    1 <runner_working_directory>/<runnerUuid>/runner.log

    For example:

    1 ~/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

    The default working directory for the runner is the /temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

  4. Review the end of the log file for errors that need to be addressed.

Updating a runner version

If you experience an issue with a runner, update the runner to the latest version.

  1. Stop the runner.

  2. Pull the latest runner version using the command below:

    1 docker image pull docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner:1
  3. Restart the runner.

Troubleshooting errors when starting a runner

An error message

Solution

docker: Error response from daemon: Conflict. The container name "/runner-76b247e7-b925-5e7b-9da2-1cda14c4ff2c" is already in use by container "c3403236e3af5962ed3a9b8771561bd2021974941cc8a89a40c6c66cecb18f53". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

Run the command docker container rm -f <runner-name>. Replace the placeholder <runner-name> with the actual name.
For example, for the error message used in this example, you would run docker container rm -f runner-76b247e7-b925-5e7b-9da2-1cda14c4ff2c command.

The installed Docker client must be at least version 19

To install a required version of the docker engine, refer to Install Docker Engine help.

No access to containers directory

Run the following command to add read permissions to the current user:
sudo chmod +r /var/lib/docker/containers

user-ns remapping not allowed in docker settings.

User-ns remapping is not supported. Change the docker daemon configuration. Refer to the following docker docs for more information: Isolate containers with a user namespace.Isolate containers with a user namespace Isolate containers with a user namespace

The docker logging driver must be json-file.

Refer to docker docs to configure a logging driver.

failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Backing filesystems without d_type support are not supported.

Refers to docker docs - use the OverlayFS storage driver.

.\start.ps1 : File C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 cannot be loaded. The file C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 is not digitally signed. You cannot run this script on the current system. For more information about running scripts and setting execution policy, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170

Allow unsigned scripts to run on Microsoft PowerShell.

Troubleshooting errors during step execution

An error message

Step execution stage

Solution

Cloning into ... fatal: unable to access ... : SSL certificate problem: self signed certificate in certificate chain

Cloning

Disable SSL verification on git clone in bitbucket-pipelines.yml

bash: ls: command not found (or any standard bash command)

Building

The command not found error mostly occurs when the PATH environment variable is configured as Workspace/Repository/Deployment variable. The default PATH where the binaries exist get overridden with the user-supplied PATH variable which is causing the error to appear in the build.

{"message":"toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"}

Pulling images locally (rate limit reached)

  • Log in to Dockerhub on the host to get a higher rate limit

AND/OR

A runner is in an unhealthy state and cannot run pipelines

Issue

The status of a runner is Unhealthy and the Runners page is showing the following message:

One or more runners are in an unhealthy state and cannot run pipelines.

Cause

At the end of each step in a pipeline, the runner cleans up build directories (folders) before running the next step. If the runner can’t perform this cleanup, it will enter an Unhealthy state. Some possible causes of this issue are:

  • The permissions for one or more files (or directories) within a build directory are preventing the runner from deleting files.

  • A program or service is accessing one or more files (or directories) in the build directory at the time of cleanup.

Solution

To find out which file or directory is preventing the runner from performing a build cleanup, review the runners' log files. Search for an error, such as:

1 An error occurred whilst tearing down directories.

A detailed error log that can be used for debugging will be displayed below this error message.

Accessing the log files

Build and services logs (not including the runners' logs) for a step are removed from a runner host once the step is complete. These logs can be found in the step logs on the Pipeline's user interface.

The location of the runner log files varies between operating systems and the type of runner.

Accessing the log files for Linux Docker runners

  1. Access the Linux device hosting the affected runner.

  2. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    1 <runner_working_directory>/<runnerUuid>/runner.log

    For example:

    1 /tmp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

    The default working directory for the runner is the /tmp directory.

  3. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on Linux

In a terminal; fix the permissions on a single file by adding write permissions for the user (u) and the users' user group (g) with the chmod command. Such as:

1 chmod ug+w <path/filename>

To fix the permissions on a directory and its files and subdirectories, add the --recursive option, such as:

1 chmod ug+w --recursive <path/filename>

The chmod command can also be appended to the script of a pipeline step if the permissions issue occurs with every pipeline executed. For example:

1 2 3 4 5 6 7 image: atlassian/default-image:3 pipelines: default: - step: name: 'Step that causes permissions issues' script: - bash ./product-build-script.sh && chmod ug+w --recursive build/

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for Linux.

Accessing the log files for Windows PowerShell runners

  1. Access the Windows device hosting the affected runner.

  2. Using PowerShell or File Explorer, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where PowerShell was working when the runner was started.
    For example: if PowerShell was working in the users' home directory, such as:

    1 PS C:\Users\Username>

    The runner installation directory would be C:\Users\Username\atlassian-bitbucket-pipelines-runner\

  3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    1 <runner_working_directory>\<runnerUuid>\runner.log

    For example:

    1 .\atlassian-bitbucket-pipelines-runner\temp\b609961f-891e-c872-c36b-f3f2c315d186\runner.log

    The default working directory for the runner is the \temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

  4. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on Windows

To fix file or directory access permissions, either:

  • Open the file Properties dialog from File Explorer and update the user permissions for the current user to Full Access for the files or folders preventing cleanup.

  • Use the PowerShell Set-Acl command to provide write and delete access to the files or folders preventing cleanup. For information on using the Set-Acl command, visit Microsoft Docs — PowerShell Set-Acl cmdlet.

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for Windows.

Accessing the log files for macOS shell runners

  1. Access the macOS device hosting the affected runner.

  2. Using Terminal or Finder, navigate to the runner installation directory (atlassian-bitbucket-pipelines-runner). This directory will be stored where Terminal was working when the runner was started.
    For example: if Terminal was working in the users' home directory (~/), such as: /Users/Username/; the runner installation directory would be /Users/Username/atlassian-bitbucket-pipelines-runner/

  3. Open the runner.log file using a text editor. This log file is stored in the runners' working directory, in a subdirectory named using the runners' UUID (a 32-character hexadecimal string). Such as:

    1 <runner_working_directory>/<runnerUuid>/runner.log

    For example:

    1 ~/atlassian-bitbucket-pipelines-runner/temp/b609961f-891e-c872-c36b-f3f2c315d186/runner.log

    The default working directory for the runner is the /temp subdirectory of the atlassian-bitbucket-pipelines-runner directory.

  4. Review the end of the log file for errors that need to be addressed.

Fixing the permissions of a file or folder on macOS

In a terminal; fix the permissions on a single file by adding write permissions for the user (u) and the users' user group (g) with the chmod command. Such as:

1 chmod ug+w <path/filename>

To fix the permissions on a directory and its files and subdirectories, add the --recursive option, such as:

1 chmod ug+w --recursive <path/filename>

The chmod command can also be appended to the script of a pipeline step if the permissions issue occurs with every pipeline executed. For example:

1 2 3 4 5 6 7 image: atlassian/default-image:3 pipelines: default: - step: name: 'Step that causes permissions issues' script: - bash ./product-build-script.sh && chmod ug+w --recursive build/

Restart the runner by re-running the command shown when setting up the runner, or recreate the runner in Bitbucket. For information on setting up the runner, see: Set up runners for macOS.

Windows PowerShell troubleshooting

Unsigned scripts in Windows PowerShell

Issue

A Windows-based runner has failed to run one or more steps due to an unsigned PowerShell (.ps1) script, resulting in an error similar to the following:

1 2 3 4 5 6 7 8 9 .\start.ps1 : File C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 cannot be loaded. The file C:\Users\Administrator\atlassian-bitbucket-pipelines-runner\bin\start.ps1 is not digitally signed. You cannot run this script on the current system. For more information about running scripts and setting execution policy, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170. At line:1 char:1 + .\start.ps1 -accountUuid '{924bbdf1-ea18-2c70-4655-2bb23075ddbf}' -re ... + ~~~~~~~~~~~ + CategoryInfo : SecurityError: (:) [], PSSecurityException + FullyQualifiedErrorId : UnauthorizedAccess

Cause

 

The Windows runner generates PowerShell scripts for cloning the repository and running the script for each step in the pipeline. These scripts are generated when the pipeline is run, preventing them from being digitally signed.

To allow the Windows runners to run unsigned PowerShell scripts, set the PowerShell execution policy of the CurrentUser to either:

  • RemoteSigned (recommended)

  • unrestricted

  • bypass

The RemoteSigned execution policy allows local unsigned (uncertified) scripts to run on the device. This includes any potentially malicious unsigned scripts. Before changing the execution policy, review the execution policies and consider their security implications at Microsoft Docs — PowerShell execution policies.

Solution

 

To check the execution policy for the CurrentUser:

  1. Open Windows PowerShell from the Windows Start menu.

  2. Run the following command, which will return the execution policy for the CurrentUser:

    1 Get-ExecutionPolicy -Scope CurrentUser

To change the execution policy for CurrentUser to RemoteSigned:

  1. In Windows PowerShell, run the following command:

    1 Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
  2. Verify that the change was successful by running Get-ExecutionPolicy and confirm that the CurrentUser has the RemoteSigned execution policy.

    1 Get-ExecutionPolicy -Scope CurrentUser

For information on Microsoft PowerShell execution policies, visit Microsoft Docs — PowerShell: about Execution Policies.

 

Additional Help