Bamboo Recovery after Encryption Key Loss

Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Problem

Bamboo stores a cipher key in the home directory (configuration/cipher) and Database (BANDANA table) which is used to decrypt sensitive data. The data that is encrypted is defined by System Wide Encryption. If the key on the filesystem is lost Bamboo will fail to start and the following occurs in the logs (atlassian-bamboo.log) as ‘Unable to read cipher data’.

//During startup
2019-05-22 17:05:28,215 FATAL [localhost-startStop-1] [InstanceSecretStorage] Filesystem part of instance key is unavailable
2019-05-22 17:05:28,217 FATAL [localhost-startStop-1] [SecretEncryptionServiceInternalImpl] 
java.lang.IllegalArgumentException: Unable to read cipher data for 0
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59)
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)...
2019-05-22 17:05:28,231 FATAL [localhost-startStop-1] [InstanceSecretStorage] Filesystem part of instance key is unavailable
2019-05-22 17:05:28,236 FATAL [localhost-startStop-1] [BambooContainer] Cannot start Bamboo
java.lang.IllegalArgumentException: Unable to read cipher data for 0
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59)
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)
...
2019-05-22 17:05:28,241 FATAL [localhost-startStop-1] [UpgradeLauncher] Failed to initialise Bamboo container
java.lang.IllegalArgumentException: Unable to read cipher data for 0
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.readKeyAndIv(InstanceSecretStorage.java:59)
	at com.atlassian.bamboo.crypto.instance.InstanceSecretStorage.loadCipherData(InstanceSecretStorage.java:116)
...
22-May-2019 17:05:28.283 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
22-May-2019 17:05:28.288 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors


//During usage with linked repositories
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalArgumentException: Cannot read keypair, it doesn't contain any line with '-----BEGIN'
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
	at com.atlassian.bamboo.utils.collections.AlwaysInvalidatingCacheDecorator.get(AlwaysInvalidatingCacheDecorator.java:65)Diagnosis

Bamboo's encryption system can't recover data if the cipher.key_0 file is lost, resulting in permanent loss of all encrypted data. This requires manually re-entering credentials, SSH keys, variables, and related information. We strongly recommend backing up the Bamboo Database before proceeding with any actions.

If we do not have a backup of the cipher keys, recovering Bamboo will be a time-consuming process. This will require us to manually recreate credentials, SSH keys, variables, and any linked plans or global repositories.

Cause

The cipher key files (cipher.key_x) located in the <Bamboo-Home>/xml-data/configuration/cipher folder on Bamboo 7 and earlier, or in the <Bamboo-Home>/shared/configuration/cipher folder on Bamboo 8 and later, are missing, failed to create or do not have the correct permissions.

Resolution

There are ways that the cipher files can be restored:

If the cipher file(s) are present, then ensure that the user who runs Bamboo has the correct permissions to access the file and restart Bamboo.
If the files are missing and a backup is available then recover the configuration/cipher/cipher.key_0 file from a backup and restore it to the cipher folder and restart Bamboo.
If the file is missing and could not be recovered from backups or has a 0 length then you can attempt the recovery procedure described in the following section.

Recovery

This section discusses the (partial) recovery of the Bamboo instance in the case the cipher.key_0 file is lost and cannot be recovered from a backup.

The design of the System-wide encryption feature of Bamboo does not allow for data recovery in case of the cipher.key_0 file is lost. All encrypted data is lost.
This procedure only cleans up encrypted data and generates new encryption keys. All encrypted data (credentials, SSH keys, variables, etc.) needs to be re-populated manually.
We highly recommend performing a backup of the Bamboo Database before proceeding with the steps below!

For the rest of the section, we will use the following notation:

BI - current Broken Bamboo Instance that will be recovered.
FI - Fresh Bamboo instance used to generate new pair of encryption keys.

1. Backup BI

Stop BI.
Save a copy of the BI install directory.
Save a copy of the BI home directory.
Create a database dump of the BI database using the native database backup tools.

2. Disable Encryption and automatic builds during startup in BI

In order to prevent BI from running automated builds, deployment and encryption during the startup, place the following arguments in the <BI-Install-dir>/bin/setenv.sh file under JVM_SUPPORT_RECOMMENDED_ARGS (see Configuring your system properties):

-Dbamboo.security.encryption.disable=true -Datlassian.bamboo.build.disable=true -Datlassian.bamboo.branch.detection.disable=true
-Datlassian.bamboo.closed.branch.detection.disable=true -Dbamboo.stash.disable.build.status.updates=true -Dbamboo.disable.stash.polling=true -Datlassian.bamboo.tag.detection.disable=true

Once complete, start the Bamboo instance.

3. List all the encrypted data in BI (database and filesystem) and update values

We perform these steps to identify which data is encrypted with the lost keys and needs to be cleaned up. Bamboo encrypts data in separate locations:

Shared Credentials with passwords.
Global / Plan variables that have been masked (*****) - These are variables that include keywords such as "secret", "password", "sshKey", and "passphrase"
Passwords and SSH private keys in Repository configurations
Mail SMTP server encryption.
ActiveMQ certificate for JMS communication between the agent and the server
Elastic agent configurations
Artifact handler credentials
User directory credentials

All data that is encrypted has a prefix of BAMSCRT. Each subsection below lists the encrypted data. After identifying the encrypted data, you need to update it with a non-encrypted string from the UI.

Always back up your data before making any database modifications. If possible, test any alter, insert, update, or delete SQL commands on a staging server first.

Please keep track of all the results/items below in a list, as you will need to reset their correct passwords after the recovery process.

3a. Shared Credentials

To find if there are any shared credentials with an encrypted password, run the following Database Query on the BI database:

SELECT * FROM credentials WHERE xml LIKE '%BAMSCRT%';

Navigate Shared credentials and update the credentials with the values with correct value or a dummy value. If you happen to have a large number of credentials, please use the statement below to update the database table. Please note that this has to be updated with the correct value after the recovery.

update credentials set xml = "DUMMY_RECOVERY_VALUE" WHERE xml LIKE '%BAMSCRT%';

3b. Global / Plan Variables

To find if there are any variables with an encrypted value, Run the following Database Query on the BI database (Tested in PostgreSQL):

SELECT variable_definition.variable_key,
variable_definition.variable_type,
build.full_key,
build.title,
variable_definition.variable_value
FROM variable_definition 
LEFT JOIN build ON (build.build_id=variable_definition.plan_id AND build.marked_for_deletion = false) 
WHERE variable_definition.variable_value LIKE '%BAMSCRT%';

Run the following query against the BI database to replace the encryption identifier "BAMSCRT" with a dummy value:

UPDATE variable_definition
SET variable_value = 'DUMMY_RECOVERY_VALUE'
WHERE variable_value LIKE '%BAMSCRT%';

Please note that this has to be updated with the correct value after the recovery.

3c. Repository Configurations

To find if there are any variables with an encrypted value, run the following Database Query on the BI database: Please note that the repository configurations has to be recreated manually during the recovery process as there is no way to replace existing encrypted data with the new values.

For Global Linked Repositories

SELECT v.NAME,
       v.is_global,
       Replace(v.plugin_key, 'com.atlassian.bamboo.plugins.', '') AS repo_type,
       v.marked_for_deletion
FROM   vcs_location AS v
WHERE  v.is_global=true
AND v.xml_definition_data LIKE '%BAMSCRT%';

For Project Repositories

SELECT v.name,
         v.is_global,
         REPLACE(v.plugin_key,
        'com.atlassian.bamboo.plugins.','') AS repo_type, 
		v.marked_for_deletion,
		PROJECT.PROJECT_KEY,
		PROJECT.TITLE AS PROJECT_TITLE
FROM vcs_location AS v
LEFT JOIN PROJECT
	ON V.PROJECT_ID=PROJECT.PROJECT_ID
WHERE v.is_global=false
AND v.project_id is not null
AND v.xml_definition_data LIKE '%BAMSCRT%';

For Plan Repositories

SELECT v.name,
         v.is_global,
         REPLACE(v.plugin_key,
        'com.atlassian.bamboo.plugins.','') AS repo_type, 
		v.marked_for_deletion, 
		build.full_key, 
		build.title AS PLAN_TITLE
FROM vcs_location AS v
LEFT JOIN plan_vcs_location
    ON v.vcs_location_id=plan_vcs_location.vcs_location_id
LEFT JOIN build
    ON (build.build_id=plan_vcs_location.plan_id
        AND build.marked_for_deletion = false)
WHERE v.is_global=false
AND v.project_id is null
AND v.xml_definition_data LIKE '%BAMSCRT%';

Generic Query

SELECT v.name,
         v.is_global,
         REPLACE(v.plugin_key,
        'com.atlassian.bamboo.plugins.','') AS repo_type, 
		v.marked_for_deletion, 
		build.full_key, 
		build.title AS PLAN_TITLE,
		PROJECT.PROJECT_KEY,
		PROJECT.TITLE AS PROJECT_TITLE
FROM vcs_location AS v
LEFT JOIN plan_vcs_location
    ON v.vcs_location_id=plan_vcs_location.vcs_location_id
LEFT JOIN build
    ON (build.build_id=plan_vcs_location.plan_id
        AND build.marked_for_deletion = false)
LEFT JOIN PROJECT
	ON V.PROJECT_ID=PROJECT.PROJECT_ID
WHERE v.xml_definition_data LIKE '%BAMSCRT%';

3d. Mail SMTP password

Unlike the other items on this list, the Mail password is stored on disk. If you have an SMTP server setup, this is located in the <BI-home-dir>/bamboo-main.cfg.xml file under the <password> tags. If you see a string prefixed with BAMSCRT either update this file or update it in the UI by navigating to:

Bamboo administration > Mail server > Edit

3e. Active MQ Certificate

If you have secured your agents following the steps inside the Securing your remote agents > Configure your Bamboo server to use SSL page by simply updating the protocol to SSL in the Broker URL and Broker Client URL inside the <bamboo-home>/bamboo.cfg.xml file Bamboo will have automatically set up the key and trust stores and distributed certificates to your agents the first time they connected to the server. This feature is called automatic keystore management.

During this process, Bamboo generates a dynamic password, which is saved in the configuration file <shared-home-dir>/bamboo-shared.cfg.xml under the jms.keystore.password tag. Since the password is generated dynamically and stored in an encrypted format, it cannot be decrypted for replacement. The only solution is to delete the broker.ks file located in <shared-home-dir>/configuration. This will prompt Bamboo to generate a new keystore automatically during the next restart. As a result of this action, agents may experience communication issues with the Bamboo server via ActiveMQ. If this issue occurs, please follow the steps outlined below on the remote agent side to reestablish the communication.

Stop agent.
Navigate to the <bamboo-agent-home>/configuration folder.
Delete the jmsclient.ks and jmsclient.ts files (or move them to another folder in the server, in case you want to have backup files).
Start agent.
1. The agent will receive the new certificate from Bamboo during startup.

Keystore password was static until Bamboo 9.2.3, and after the vulnerability, Bamboo has started generating a dynamic password for the keystore.
Steps for regenerating the broker.ks file is documented in the KB article How to generate a new Bamboo ActiveMQ broker certificate using automatic keystore management | Bamboo | Atlassian Support .
There is a feature request BAM-21721 open to avoid the deletion of jmsclient.ks and jmsclient.ts files once the broker.ks is regenerated.

3f. Elastic Agent Configuration

If you have an enabled Elastic Agents setup, this is located in the <shared-home-dir>/configuration/administration.xml file under the <elasticConfig> tags. If you see a string prefixed with BAMSCRT either update this file or update it in the UI by navigating to:

Bamboo administration > Elastic Bamboo > Configuration > Edit

3g. Artifact handler Configuration

If you have configured Artifact Handlers like Amazon S3 or SFTP Artifact handlers, then the authentication details, such as the secret access key, SSH Private Keys, and password, will be saved in encrypted format.

select serialized_data from public.bandana where bandana_key='artifactHandlersConfiguration';

If you see a string prefixed with BAMSCRT, update it in the UI by navigating to:

Bamboo administration > Artifact handler > Edit

3h. User Directory Configuration

If you have integrated Bamboo with an external User Directory like Atlassian Crowd, Atlassian JIRA, Microsoft Active Directory, or LDAP, then the password for authenticating the user directory will be saved in an encrypted format. To handle this, you can choose one of the options below:

Option 1: Use the below database query to identify a string prefixed with BAMSCRT, update it in the UI for each configured user directory by navigating to:

Bamboo administration > Security > User directories > Edit

select CD.DIRECTORY_NAME,CDA.ATTRIBUTE_VALUE from CWD_DIRECTORY_ATTRIBUTE CDA
INNER JOIN CWD_DIRECTORY CD
	ON CDA.DIRECTORY_ID=CD.ID
where CDA.attribute_name like '%password';

This option can be used especially during any of the following cases:

You are using Bamboo 10.0.0 or later.
You are fine to restart the Bamboo application multiple times.
You don't know the list of permissions that are managed in Bamboo for the users and groups synced from the user directory.

This is the best option if you are using Bamboo 10.0 or later. If you are using a version lower than Bamboo 10.0, this option can also be used, but you will need to follow a few additional steps at a later stage.

Option 2: Record all current configurations and permissions for users and groups synced from the user directory before proceeding with this option. Delete the user directory, which will remove all synced users, groups, and their associated permissions from the system.

This option can be used especially during any of the following cases:

You are using a version lower than Bamboo 10.0.
You don't want to restart the Bamboo application multiple times.
You know the list of permissions that are managed in Bamboo for the users and groups synced from the user directory.

Based on the options you have chosen, please follow the steps as guided once the instance is restored.

4. Cleanup encrypted data in BI

If the <BI-Home>/bamboo-main.cfg.xml does not contain an SMTP server with a <password> value defined, backup up the original file and replace the contents with:

<?xml version="1.0" encoding="UTF-8"?>

<mail-servers>
  <smtp-server>
    <name>Bamboo</name>
    <password>password</password>
    <defaultFrom>noreply@test.com</defaultFrom>
    <prefix>Bamboo</prefix>
    <hostname>bamboo</hostname>
    <sessionServer>false</sessionServer>
    <port>25</port>
    <tlsRequired>false</tlsRequired>
  </smtp-server>
</mail-servers>

All the remaining data listed in section 3 will be cleaned up in the Bamboo UI as shown in section 6.

5. Generate new encryption keys in FI

To generate new encryption keys you will need to use another fresh installation (FI) of Bamboo. These new keys will overwrite the keys that were lost in the Broken Instance (BI).

Install a new Bamboo instance with the same version of BI (using different home and install directories).
To force the initial key generation, create a global variable with the Variable name containing 'password' in it and a dummy value and 'add'.
Stop FI
Extract the encryption keys from this FI. The below query in the FI database should list the database part of the encryption keys.
SELECT * FROM BANDANA WHERE BANDANA_KEY LIKE '%cipher%';
The file system part of the key is located in <FI-home-dir>/xml-data/configuration/cipher

We will use these in the next section to update the broken instance.

6. Finalize the recovery stage

6a.Replace the cipher key in the corrupt installation with the regenerated cipher key from FI.

Run the below queries to update encryption keys (Tested this query in PostgreSQL).
UPDATE bandana SET serialized_data = '<string>ReplaceThisWithKeyStringFromFI</string>' WHERE bandana_key='com.atlassian.restricted.instance.cipher.key_0'; UPDATE bandana SET serialized_data = '<string>ReplaceThisWithIVStringFromFI</string>' WHERE bandana_key='com.atlassian.restricted.instance.cipher.iv_0';
copy the cipher file from <FI-home-dir>/xml-data/configuration/cipher to <BI-home-dir>/xml-data/configuration/cipher

6b. Start BI with encryption disabled

Ensure that the Bamboo instance starts correctly. If Bamboo fails to start with errors related to cipher data, check for any entries that were missed from the previous section (Identify affected data and update it if necessary).

6c. Update the rest of the previously encrypted data in BI.

Now that we have BI running, we will update the passwords listed in section 3.

Navigate and update manually Shared Credentials and Global / Plan Variables in BI.
Navigate and update Repository Configurations
- If the repository is using a username/password, you need to set the correct password.
- If the repositories are using SSH keys,
  1. You need to re create the Linked repository again, and using the DB queries, you can find out the details about repos and their usage to link them back to the plans : Extract Repository details from the Bamboo database | Bamboo | Atlassian Support

7. Re-enable encryption in BI

Now that we have repopulated all the credentials, we restart BI with encryption enabled, so that BI will encrypt all credentials with the new keys.

Stop BI
Remove the argument -Dbamboo.security.encryption.disable=true from the setenv.sh file (set in section 2)
Start Bamboo

After the BI is available, all secrets will remain in plain text format that were updated earlier. At this point, we strongly recommend that you review each secret that you updated earlier and save it again through the Bamboo UI. This process ensures that the secrets are encrypted using cipher keys generated by the FI.

If you integrate with user directories, please follow the steps below based on the Option you have chosen in Step 3h.

Option 1:

If you are using Bamboo 10.0 or later, the Bamboo instance will come up normally.

If you are using a version that is lower than Bamboo 10.0, it will not come up, even though the password was updated as part of Step 3h. This is because when encryption is enabled, Bamboo can't understand the password saved when encryption is disabled. Please follow the steps below to restore the instance.

Stop BI
Place the following arguments in the <BI-Install-dir>/bin/setenv.sh file under JVM_SUPPORT_RECOMMENDED_ARGS (see Configuring your system properties):

-Dbamboo.security.decryption.ignore.errors=true

Start Bamboo
Navigate to User Directories and update the password once again again from the Bamboo UI.
Stop BI
Remove the argument -Dbamboo.security.decryption.ignore.errors=true
Start Bamboo

Option 2: Recreate the user directories using the previous configurations and reassign the necessary permissions to users and groups. This approach is straightforward and ensures a clean restoration process.

Please check Bamboo application now started without any encryption errors and system seems healthy you can now remove the below parameters from JVM_SUPPORT_RECOMMENDED_ARGS and restart for Bamboo application to function as normal with automatic builds and deployments.

-Datlassian.bamboo.build.disable=true,-Datlassian.bamboo.branch.detection.disable=true
-Datlassian.bamboo.closed.branch.detection.disable=true,-Dbamboo.stash.disable.build.status.updates=true, -Dbamboo.disable.stash.polling=true

The instance is now recovered. You may now remove the Fresh Instance (FI) from your system.

If you require any further support with these steps please contact our friendly support.

Updated on October 16, 2025

Was this helpful?

It wasn't accurateIt wasn't clearIt wasn't relevant