Set up runners for Windows

Windows Runners allow you to run builds in Pipelines on your own Windows infrastructure, and you won’t be charged for the build minutes used by your self-hosted windows runners.

Prerequisites

Below are the sample chocolatey scripts to install all of the prerequisites required:

1 2 3 4 choco install -y git choco install -y openjdk11 choco install -y dotnetfx --pre choco install git-lfs.install # if you need to use git-lfs features

To improve build times, we recommend installing any other dependencies your Pipelines require in advance, such as nuget, xUnit, nUnit, etc.

Allow unsigned scripts to run in PowerShell

The Windows runner generates PowerShell scripts for cloning the repository and running the script for each step in the pipeline. These scripts are generated when the pipeline is run, preventing them from being digitally signed.

To allow the Windows runners to run unsigned PowerShell scripts, set the PowerShell execution policy of the CurrentUser to either:

  • RemoteSigned (recommended)

  • unrestricted

  • bypass

The RemoteSigned execution policy allows local unsigned (uncertified) scripts to run on the device. This includes any potentially malicious unsigned scripts. Before changing the execution policy, review the execution policies and consider their security implications at Microsoft Docs — PowerShell execution policies.

To check the execution policy for the CurrentUser:

  1. Open Windows PowerShell from the Windows Start menu.

  2. Run the following command, which will return the execution policy for the CurrentUser:

    1 Get-ExecutionPolicy -Scope CurrentUser

To change the execution policy for CurrentUser to RemoteSigned:

  1. In Windows PowerShell, run the following command:

    1 Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
  2. Verify that the change was successful by running Get-ExecutionPolicy and confirm that the CurrentUser has the RemoteSigned execution policy.

    1 Get-ExecutionPolicy -Scope CurrentUser

For information on Microsoft PowerShell execution policies, visit Microsoft Docs — PowerShell: about Execution Policies.

Disable the Windows pagefile and swapfile

Best practice

Before you create a Windows Runner, we strongly recommend disabling swapfile.sys and pagefile.sys in your Windows environment. Having swap enabled can lead to non-deterministic build results in regards to memory and OOMing, meaning that sometimes enough swap is available and a build may pass, while other times not enough swap is available which could make the same build OOM.

Follow the steps below to disable pagefile.sys and swapfile.sys in Windows 10. If the following instructions do not work, consult your distributions documentation to configure your Windows environment:

  1. In Windows, select Start, type Advanced System Settings into the Start menu and press Enter to open it

  2. Select the Advanced tab and then the Settings button in the Performance section of the System Properties dialog.

  3. Select the Advanced tab and then the Change button in the Virtual memory section of the Performance Options dialog.

  4. Unselect Automatically manage paging file size for all drives and select No paging file in the Page file size for each drive section of the Virtual Memory dialog, and then select the Set button.

  5. Select OK and then reboot your system.

Using your runner

Currently we only support running one runner per machine.

  1. Download the zip file provided in Run step on the Runner installation dialog.

  2. Unzip the zip file to the desired directory, for example: C:\Users\your_user_name\atlassian_runners

  3. Open PowerShell as an administrator, go to the bin directory under your Runner folder, run the command provided in Run step on the Runner installation dialog.

  4. Windows Runners is using PowerShell and will run directly in your instance which means it is not a clean build environment for every step, and any side effects generated by the step (i.e., installing any applications, starting a db service, or editing a file outside of the build directory) would potentially affect the next step to be run. Because of this, the runner would only make a best effort to keep the build directory empty. It is your responsibility to make sure the script you run in your step won’t have a major impact on other steps.


Limitations for Windows Runner

Shared build environment

The Windows Runner uses PowerShell to execute the step scripts, and the host machine will be shared by multiple steps that are scheduled to execute on the runner. If you install/change something globally in your step, such as installing a new library, then this change will affect the next step run on the host machine.

Unsupported features

The following are the features that won't be support by self-hosted Windows Runners due to limitations on how Windows Runners is implemented and security complications:

  • SSH keys: For more information, refer toVariables and secrets

  • Pipes: For more information, refer toWhat are pipes?

  • Service containers: For more information, refer toDatabases and service containers

  • Custom build images on AWS ECR

  • Custom Docker in Docker image

  • Step size: Given that Windows Runners runs directly on your Windows instance, we don’t put any restriction on memory usage.

Limitations and workarounds

Cache

  • Pre-defined Docker cache is not supported: Given that Docker is not supported for Windows Runners, pre-defined Docker cache is also not supported in Windows Runners.

  • Share caches between different OS: As a best practice, we recommend specifying different cache name for different Runner type, such as a Linux runner and a Windows runner, because caches could contain platform specific files that do not work in other operating systems. This might lead to errors occurring when a Windows runner is trying to use a .dll file that is specifically generated for Linux.

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 pipelines: custom: customPipelineWithRunnerAndCache: - step: name: Step 1 runs-on: - 'windows' script: - echo "This step will run on a self hosted windows infrastructure."; caches: - windows_bundler - step: name: Step 2 runs-on: - 'linux' script: - echo "This step will run on a self hosted linux infrastructure."; caches: - linux_bundler - step: name: Step 3 runs-on: - 'linux' script: - echo "This step will run on Atlassian's infrastructure as usual."; caches: - linux_bundler definitions: caches: linux_bundler: vendor/bundle windows_bundler: vendor/bundle
  • Bloated cache folder: Due to performance implications, we do not clean up the cache folder at the end of step execution which means the size of related cache directories could increase rapidly, especially for a workspace runner. If that occurs, we recommend creating a scheduled task to clean up cache folders on a regular basis. You can refer to following documentation to see how to setup a scheduled task with PowerShell: https://docs.microsoft.com/en-us/powershell/module/scheduledtasks/get-scheduledtask?view=windowsserver2019-ps.

  • Be aware that we don’t put any restrictions on where your cache folder is located, so technically you can define whatever folder you like, even c:\windows. Be mindful about any technical implications and make sure your hosted machine is recoverable.

Test Reporting

There is some additional setup required for dotnet test reporting, refer to the following support document for details: Test reporting in Pipelines

Git LFS

In order to use Git LFS, you need to make sure you have Git LFS installed on your hosted machine. If you use chocolatey, then you can use the following script to setup Git LFS:

1 2 choco install git-lfs.install git lfs install

Conditional step

The glob path defined in the step condition can only support a forward slash (/) and not a backslash (\) even if the step runs on Windows. So it would look like the example provided below:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 - step: name: step1 runs-on: - self.hosted - windows script: - echo "failing paths" - exit 1 condition: changesets: includePaths: # only xml files directly under path1 directory - "path1/*.xml" # any changes in deeply nested directories under path2 - "path2/**"

SSH Keys

You'll want to set up an SSH key in Bitbucket Pipelines if:

  • your build needs to authenticate with Bitbucket or other hosting services to fetch private dependencies.

  • your deployment needs to authenticate with a remote host or service before uploading artifacts.

  • you want builds to use tools such as SSH, SFTP, or SCP.

However, due to security reasons, a Windows Runner will not add your SSH keys to the build environment automatically like a cloud runner does. But you are still able to do it manually if you want to, below are the example steps to add your SSH key manually with OpenSSH:

  • Generate an SSH key (if necessary) 
    You can run the following in a terminal to generate an RSA key pair:

    1 $ ssh-keygen -t rsa -b 4096 -N '' -f my_ssh_key
  • Encode a private key with base64
    Pipelines does not currently support line breaks in environment variables, so you can encode the private key with base64 by running the following:

    1 [convert]::ToBase64String((Get-Content -path "my_ssh_key" -Encoding byte))
  • Add the key as a secure variable
    Copy the encoded key from the terminal and add it as a secured Bitbucket Pipelines environment variable to the repository:

    1. In the Bitbucket repository, select Repository settings > Repository variables.

    2. Copy the base64-encoded private key from the terminal.

    3. Paste the encoded key as the value for an environment variable. Make sure to check Secured.

There are security risks associated with passing private SSH keys as repository variables:

  • Repository variables get copied to child processes that your pipelines build may spawn.

  • Secured variables can be retrieved by all users with write access to a repository.

We recommend that you never pass your own personal SSH key as a repository variable but instead, generate a new SSH key-pair for Pipelines that easily be disabled if it is compromised. It may also be worth using deployment variables, which you can combine with deployment permissions to control access. Learn more about deployment variables.

  • Install the public key on a remote host
    You must install the public key on the remote host before Pipelines can authenticate with that host. If you want your Pipelines builds to be able to access other Bitbucket repositories, you need to add the public key to that repository.

    Remote hosts
    If you have SSH access to the server, you can use the ssh-copy-id  command. Typically, the command appends the key to the ~/.ssh/authorized_keys file on the remote host:

    1 ssh-copy-id -i my_ssh_key username@remote_host

    Test the SSH access to the server:

    1 ssh -i ~/.ssh/my_ssh_key user@host
  • Get the host keys and add them to ~/.ssh/known_hosts file in the host virtual machine (VM).
    The known_hosts file contains the DSA host keys of SSH servers accessed by the user. It's important to verify that you're connecting to the correct remote host.

    1. Get the DSA host keys of any remote servers. You can do this by executing the following command:

      1 ssh-keyscan -t rsa server.example.com
    2. Add those keys to the ~/.ssh/known_hosts file in the host VM. You can remove any unrelated lines.

  • Tie everything together in the bitbucket-pipelines.yml file (as shown in the example below):

    1 2 3 4 5 6 7 image: node:6 # specify your Docker image here pipelines: default: - step: script: - (umask 077 ; echo $MY_SSH_KEY | base64 --decode > ./id_rsa) - ssh -i ./id_rsa <user>@<host> 'echo "connected to `host` as $USER"'

In the script provided above, we use ./id_rsa instead of ~/.ssh/another_private_key. This ensures that the Windows Runner will monitor the file that gets generated in the runner build folder and will do its best to clean it up at the end of the step. Any files that are created outside of the runner build folder will not be removed and leaving private keys in ~/.ssh will increase the chance of the key get exploited.

There is still a chance that we will not able to clean up the build folder. We suggest you update the SSH key-pair used in the step on a regular basis to reduce the chance of any of your data being compromised.

Build folder not being cleaned up properly

At the end of a step, the runner will attempt to clean up the build folder. This is a best effort cleanup and in some cases runners are not able to clean up the build folder. The following are the two most common cases that prevent a runner from cleaning up the build folder:

  • The step script creates files that runners do not have permission to remove, for example, set FullControl as Deny for all users.

  • The step script is creating orphaned processes that a runner is not able to stop, and some of the files in the build are locked by those orphaned processes. For example, a hosted machine had Gradle daemon (which is not suggested by Gradle), and is set to use daemon for all Gradle builds by: org.gradle.daemon=true. In this case, the processes created by Gradle builds will attach to the existing daemon process, and the runner would not able to stop them at the end of the step.

Any of the above cases will prevent the runner from cleaning up the build folder, and will cause the step to fail with an error informing you to check the runner logs and manually clean up the build folder in the host machine. Any following steps that are scheduled to run on this runner will also cause an error because the build folder was not cleaned up properly.

Solution: You need to contact the admin who has access to the runner and check the runner logs to find out which files are preventing the runner from cleaning up the build folder.

For files that have permissions to prevent runners from removing them, the admin needs to fix the permissions for those files, and then remove the build folder and restart runner.

For files that are locked by orphaned processes, the admin needs to find out the process that locked the files, and kill the process. Then remove the build folder and restart runner.

For either of the above solutions, you need to revisit the step script and the current settings for the host machine which will ensure the step won’t lock the build folder next time you run the step, or you will have to do the manual clean up in host machine again. In the case of Gradle daemon, you can add the --no-daemon option in the Gradle build, and it will prevent any child processes attached to the Gradle daemon.

Additional Help