"This secret cannot be decrypted" error in Bitbucket Data Center

Summary

The application link, mail notification, and the search features fail in Bitbucket Data Center. The error message that appears is:

"com.atlassian.secrets.api.SecretServiceException:This secret cannot be decrypted with the configured encryption key."

Concurrently, the error 'LDAP: error code 49 - Invalid Credentials' is also present, leading to failures in external user logins and directory synchronization.

Environment

Bitbucket Data Center 9.4.9, but may apply to other versions.

Diagnosis

There are four common scenarios that will point to this error. They are detailed below.

Scenario 1

The atlassian-bitbucket.log will display the following error when an external directory user attempts to log in to Bitbucket or when the user clicks the synchronization button for the external directory. This may also occur in the situation if <Bitbucket-Home-Directory>/shared/keys/javax.crypto.spec.SecretKeySpec_<unixtimestamp> file is deleted from the fileserver.

Scenario 2

The atlassian-bitbucket.log will show the following error when Bitbucket attempts to connect to the Search process.

Scenario 3

The atlassian-bitbucket.log will show the following error when Bitbucket tries to send an email to the mail server. Upon reviewing the Settings > Mail server page, the password field is blank, despite the mail server configuration containing a password before this issue occurred.

Scenario 4: Existing application links will show a 'Network error' from the Bitbucket side while appearing connected from the Jira/Bamboo side, and the atlassian-bitbucket.log will show the following error:

If you attempt to recreate the application links, a "Config Error" will be displayed. When editing the same application link in Bitbucket, an error message states, "The local outgoing connection uses Auth, however it's disabled on <app-link-name>." Attempting to change the status from disabled to OAuth on the Jira/bamboo side and saving the app link is not permitted, fails with error message: "We can't enable the incoming authentication because <Bitbucket-app-link-name> is not reachable". The atlassian-bitbucket.log will also contain the following error.

Cause

These errors usually arise when Bitbucket's database is overwritten, while the data in the shared home directory on the filesystem is not up-to-date.

Bitbucket securely stores the user directory, mail, search server passwords, and OAuth consumer secrets in the database, while the corresponding encrypted keys are kept in the filesystem. If the database from another Bitbucket instance or from an older backup, which does not share the same key, is overwritten and the shared home directory data is not updated accordingly, the errors may appear.

Solution

For Test instance:

If this is the test instance, the easiest and straightforward approach is to refresh the entire data of staging from prod by following How to setup staging or test server environments for Bitbucket Data Center article. Doing so will not only resolve all these issues but also ensure you have an exact replica of the production environment, allowing you to assess how changes will perform in a production-like setting accurately.

For Production instance:

If it's a production environment, roll back the instance using the backup taken prior to the activity, as these issues are not the only ones that may arise. However, if a rollback isn't feasible or if these errors are due to a different issue, attempt a resolution specific to the scenario. However, this doesn’t mean that it will fix other errors and issues of the Bitbucket instance.

• Scenario 1 Resolution: Log in with a local user account that has sysadmin privileges, navigate to Settings > User Directories, edit the user directory, enter the correct password in the Password field, and save it. This will generate a new file named <Bitbucket-Home-Directory>/shared/keys/javax.crypto.spec.SecretKeySpec_<unixtimestamp>, which will re-encrypt the directory password. As a result, directory synchronization and external user login will begin functioning again.

• Scenario 2 Resolution: Log in with a user account that has sysadmin privileges(if external users are not working, log in with a local user), go to Settings > Server settings, enter the correct password in the 'Search server password' field, test it, and save it. If you are using a bundled search and the password is unknown, refer to "Access Denied" when testing connection to search server in Bitbucket Data Center for all the necessary steps.

• Scenario 3 Resolution: Log in with a user account that has sysadmin privileges(if external users are not working, log in with a local user), navigate to Settings > Mail server, enter the correct password in the Password field, and save it. Test the feature by sending a test email.

• Scenario 4 Resolution: Take a complete backup of the Bitbucket instance. Please refer to the Data recovery and backups article. Do not proceed without taking a backup. Additionally, make a separate backup of the bb_sealed_secret table from the Bitbucket database.

1. Shut down all Bitbucket applications in the cluster.

2. Delete the row with identifier 'bitbucket.secrets.external.com.atlassian.oauth.consumer.ConsumerService:host.__HOST_SERVICE__' from the bb_sealed_secret table:

3. delete from bb_sealed_secret where bb_sealed_secret.identifier like 'bitbucket.secrets.external.com.atlassian.oauth.consumer.ConsumerService:host.__HOST_SERVICE__';

4. Start a Bitbucket application and recreate the application link.

5. Verify that the application link is functioning properly and that other Bitbucket functionalities work as expected. Start the Bitbucket applications on the remaining nodes.