Bamboo Recovery after Encryption Key Loss

Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

Bamboo stores a cipher key in the home directory which is used to decrypt sensitive data. The data that is encrypted is defined by System Wide Encryption.

If the key on the filesystem is lost Bamboo will fail to start and the following occurs in the logs (catalina.out):

//During startup
2019-05-22 17:05:28,215 FATAL [localhost-startStop-1] [InstanceSecretStorage] Filesystem part of instance key is unavailable
2019-05-22 17:05:28,217 FATAL [localhost-startStop-1] [SecretEncryptionServiceInternalImpl] 
java.lang.IllegalArgumentException: Unable to read cipher data for 0
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59)
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)...
2019-05-22 17:05:28,231 FATAL [localhost-startStop-1] [InstanceSecretStorage] Filesystem part of instance key is unavailable
2019-05-22 17:05:28,236 FATAL [localhost-startStop-1] [BambooContainer] Cannot start Bamboo
java.lang.IllegalArgumentException: Unable to read cipher data for 0
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59)
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)
...
2019-05-22 17:05:28,241 FATAL [localhost-startStop-1] [UpgradeLauncher] Failed to initialise Bamboo container
java.lang.IllegalArgumentException: Unable to read cipher data for 0
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59)
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)
...
22-May-2019 17:05:28.283 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
22-May-2019 17:05:28.288 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors


//During usage with linked repositories
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Cannot read keypair, it doesn't contain any line with '-----BEGIN'
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
	at com.atlassian.bamboo.utils.collections.AlwaysInvalidatingCacheDecorator.get(AlwaysInvalidatingCacheDecorator.java:65)Diagnosis

Cause

The cipher key files (cipher.key_x) located in the <Bamboo-Home>/xml-data/configuration/cipher folder on Bamboo 7 and earlier, or in the <Bamboo-Home>/shared/configuration/cipher folder on Bamboo 8 and later, are missing, failed to create or do not have the correct permissions.

Resolution

There are 3 states that the cipher files can be:

If the cipher file(s) are present then ensure that the user that runs Bamboo has the correct permissions to access the file and restart Bamboo
If the files are missing and a backup is available then recover the cipher.key_0 file from a backup and restore it to the cipher folder and restart Bamboo
If the file is missing or has a 0 length then you can attempt the recovery procedure described in the following section

Recovery

This section discusses the (partial) recovery of the Bamboo instance in the case the cipher.key_0 file is lost and cannot be recovered from a backup. For the rest of the section, we will use the following notation:

BI - Broken Bamboo Instance that will be recovered.
FI - Fresh Bamboo instance used to generate new encryption keys.

The design of the System-wide encryption feature of Bamboo does not allow for data recovery in case of the cipher.key_0 file is lost. All encrypted data is lost.
This procedure only cleans up encrypted data and generates new encryption keys. All data that was encrypted (credentials, SSH keys, variables, etc.) needs to be re-populated manually.
We highly recommend performing a backup before proceeding with the steps below!

1. Backup BI

Stop BI.
Save a copy of the BI install directory.
Save a copy of the BI home directory.
Create a database dump of the BI database using the native database backup tools.

2. Disable Encryption and automatic builds during startup in BI

In order to prevent BI from running automated builds during the startup, place the following arguments in the <BI-Install-dir>/bin/setenv.sh file under JVM_SUPPORT_RECOMMENDED_ARGS (see Configuring your system properties):

-Dbamboo.security.encryption.disable=true -Datlassian.bamboo.build.disable=true

3. List all the encrypted data in BI (database and filesystem)

We perform these steps to identify which data is encrypted with the lost keys and needs to be cleaned up. Bamboo encrypts data in 4 separate locations:

Shared Credentials with passwords.
Global / Plan variables that have been masked (*****) - These are variables that include keywords such as "secret" and "password"
Passwords and SSH private keys in Repository configurations
Mail SMTP server encryption.

All data that is encrypted has a prefix of BAMSCRT. Each subsection below lists the encrypted data. After Identifying the encrypted data, you need to update it with a non-encrypted string from the UI.

Always back up your data before making any database modifications. If possible, test any alter, insert, update, or delete SQL commands on a staging server first.

Please keep track of all the results/items below in a list, as you will need to reset their correct passwords after the recovery process.

3a. Shared Credentials

To find if there are any shared credentials with an encrypted password, run the following Database Query on the BI database:

SELECT * FROM credentials WHERE xml LIKE '%BAMSCRT%';

3b. Global / Plan Variables

To find if there are any variables with an encrypted value, Run the following Database Query on the BI database (Tested in PostgreSQL):

1
2
3

SELECT variable_definition.variable_key, variable_definition.variable_type, build.full_key, build.title, variable_definition.variable_value FROM variable_definition
LEFT JOIN build ON (build.build_id=variable_definition.plan_id AND build.marked_for_deletion = false)
WHERE variable_definition.variable_value LIKE '%BAMSCRT%';

3c. Repository Configurations

To find if there are any variables with an encrypted value, run the following Database Query on the BI database:

SELECT v.name, v.is_global, REPLACE(v.plugin_key,'com.atlassian.bamboo.plugins.','') as repo_type, v.marked_for_deletion, build.full_key, build.title FROM vcs_location AS v
LEFT JOIN plan_vcs_location ON v.vcs_location_id=plan_vcs_location.vcs_location_id
LEFT JOIN build ON (build.build_id=plan_vcs_location.plan_id AND build.marked_for_deletion = false)
WHERE v.xml_definition_data LIKE '%BAMSCRT%';

3d. Mail SMTP password

Unlike the other items on this list, the Mail password is stored on disk. If you have an SMTP server setup, this is located in the <BI-home-dir>/bamboo-main.cfg.xml file under the <password> tags. If you see a string prefixed with BAMSCRT either update this file or update it in the UI by navigating to:

Bamboo administration > Mail server > Edit

4. Cleanup encrypted data in BI

In this section, we clean up the encrypted repository entries that are listed in section 3c.

Run the following query against the BI database to replace the encryption identifier "BAMSCRT" with a dummy value:

UPDATE vcs_location SET xml_definition_data = REPLACE(xml_definition_data,'BAMSCRT','-----BEGIN KEY----- OLD -----END KEY-----') WHERE xml_definition_data LIKE '%BAMSCRT%';

Note: The replacement value must be "-----BEGIN KEY----- OLD -----END KEY-----" to recover certain types of repositories

Run the vcs_location select query (from 3c) again to confirm no variables are left, keeping the original output for later.

If the <BI-Home>/bamboo-main.cfg.xml does not contain an SMTP server with a <password> value defined, backup up the original file and replace the contents with:

<?xml version="1.0" encoding="UTF-8"?>

<mail-servers>
  <smtp-server>
    <name>Bamboo</name>
    <password>password</password>
    <defaultFrom>noreply@test.com</defaultFrom>
    <prefix>Bamboo</prefix>
    <hostname>bamboo</hostname>
    <sessionServer>false</sessionServer>
    <port>25</port>
    <tlsRequired>false</tlsRequired>
  </smtp-server>
</mail-servers>

All the remaining data listed in section 3 will be cleaned up in the Bamboo UI as shown in section 6.

5. Generate new encryption keys in FI

To generate new encryption keys you will need to use another fresh installation (FI) of Bamboo. These new keys will overwrite the keys that were lost in the Broken Instance (BI).

Install a new Bamboo instance with the same version of BI (using different home and install directories).
To force the initial key generation, create a global variable with the Variable name containing 'password' in it and a dummy value and 'add'.
Stop FI
Extract the encryption keys from this FI. The below query in the FI database should list the database part of the encryption keys.
1SELECT * FROM BANDANA WHERE BANDANA_KEY LIKE '%cipher%';
The file system part of the key is located in <FI-home-dir>/xml-data/configuration/cipher

We will use these in the next section to update the broken instance.

6. Finalize the recovery stage

6a.Replace the cipher key in the corrupt installation with the regenerated cipher key from FI.

Run the below queries to update encryption keys (Tested this query in PostgreSQL).
1 2 3 4 5UPDATE bandana SET serialized_data = '<string>ReplaceThisWithKeyStringFromFI</string>' WHERE bandana_key='com.atlassian.restricted.instance.cipher.key_0'; UPDATE bandana SET serialized_data = '<string>ReplaceThisWithIVStringFromFI</string>' WHERE bandana_key='com.atlassian.restricted.instance.cipher.iv_0';

copy the cipher file from <FI-home-dir>/xml-data/configuration/cipher to <BI-home-dir>/xml-data/configuration/cipher

6b. Start BI with encryption disabled

Ensure that the Bamboo instance starts correctly. If Bamboo fails to start with errors related to cipher data, check for any entries that were missed from the previous section (Identify affected data and update it if necessary)

6c. Update the rest of the previously encrypted data in BI.

Now that we have BI running, we will update the passwords listed in section 3.

Navigate and update manually Shared Credentials and Global / Plan Variables in BI.
Navigate and update Repository Configurations using the output of the query in section 3c:

1. If "marked_for_deletion" is "true"
  1. These items can be ignored, as they are only used when re-running old builds
2. If "is_global" is "true"
  1. Navigate to Bamboo administration > Linked Repositories
  2. Select the Repository that requires an update
  3. Check "Change Password?" and set the correct password
  4. Save
3. If "is_global" is "false"
  1. Determine the Plan where the repository is located, defined by the "full_key" and "title" columns
  2. Navigate to Plans > All Build Plans > and configure the plan shown from the Query
  3. Select the "Repositories" tab and select the repository that requires an update
  4. Check "Change Password?" and set the correct password
  5. Save
4. If "type" starts with "stash.atlassian-bamboo-plugin-stash"
  1. Use steps b. or c. to navigate to the repository in the UI
  2. You should see an error such as: "Could not access this repository in Bitbucket Server"
  3. Press "Save Repository" to clear the error and regenerate a new encrypted SSH key

7. Re-enable encryption in BI

Now that we have repopulated all the credentials, we restart BI with encryption enabled, so that BI will encrypt all credentials with the new keys.

Stop BI
Remove the argument -Datlassian.bamboo.build.disable=true -Dbamboo.security.encryption.disable=true from the setenv.sh file (set in section 2)
Start Bamboo

The instance is now recovered. You may now remove the Fresh Instance(FI) from your system.

If you require any further support with these steps please contact our friendly support.

Updated on April 16, 2025

Was this helpful?

It wasn't accurateIt wasn't clearIt wasn't relevant

Atlassian Support