"Access Denied" when testing connection to search server in Bitbucket Data Center

Summary

During the installation of Bitbucket, Bitbucket will hash a random password, which will be used between Bitbucket and bundled OpenSearch for authentication. This password will be stored in the Bitbucket database and OpenSearch security index.

Due to the nature of the provisioning, the password is unknown. If this goes out of sync between Bitbucket and bundled OpenSearch, we will need to set the password for bundled OpenSearch manually.

Environment

• Bitbucket 8 with bundled OpenSearch v1

• Bitbucket 9 with bundled OpenSearch v2

Diagnosis

1. Navigate to Administration > Server Settings.

2. Click on the Test button under the Search.

3. The result would be "Access to the search server was denied. Check your settings.".

4. During startup, Bitbucket application log; $BITBUCKET_HOME/log/atlassian-bitbucket.log , shows the following error messages:

   1 2 3 4 5 2025-03-04 03:14:16,621 INFO [Caesium-1-1] c.a.b.i.s.i.jobs.StartupChecksJob Running startup jobs for search 2025-03-04 03:14:17,793 ERROR [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Failed to create indexes: ''[Unsuccessful search response: 401. Bitbucket did not have authorization to request data from the search server.]'' 2025-03-04 03:14:17,793 ERROR [Caesium-1-1] c.a.b.i.s.i.IndexingSynchronizationService Unable to synchronize the mapping in the search server 2025-03-04 03:14:17,794 WARN [Caesium-1-1] c.a.b.i.s.c.cluster.ClusterJobRunner Job execution failed. Rescheduling. Job key: com.atlassian.bitbucket.internal.search.indexing.jobs.StartupChecksJob Job id: c2ffe4a6-5d69-4e4e-8d6b-fda3afd54010 ...

5. Bundled OpenSearch log; $BITBUCKET_HOME/log/search/bitbucket_search.log , show authentication warnings:

   1 2 [2025-03-04T03:14:17,384][WARN ][o.o.s.a.BackendRegistry ] [bitbucket_bundled] Authentication finally failed for bitbucket from 127.0.0.1:57880 [2025-03-04T03:14:17,774][WARN ][o.o.s.a.BackendRegistry ] [bitbucket_bundled] Authentication finally failed for bitbucket from 127.0.0.1:57880

Cause

The password expected by the Search server is out of sync with the one that Bitbucket uses to authenticate.

Solution

INFO

• Please make sure to have a backup of the $BITBUCKET_HOME/shared/search before proceeding.

• Please replace the $BITBUCKET_INSTALL and $BITBUCKET_INSTALL accordingly.

• For Bitbucket 8 (bundled with OpenSearch v1), the path is

  1 $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml

  instead of

  1 $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml

  This is due to the changes in OpenSearch v2.

1. Log in to the Bitbucket machine (SSH and any remote connection).

2. Stop Bitbucket. ( Check to see if the search process is down. You can do this by running the Linux command: ps -ef | grep java. If the search service is still running, use kill -9 <PID> to terminate it )

3. Hash a new password by running the following command:

   1 2 3 4 # /bin/bash +x $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/tools/hash.sh WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java [Password:] <qwerty> $2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC

4. Update the hash value in $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml.

   1 2 3 4 5 6 7 8 9 10 11 # vi $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml # cat $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml _meta: type: "internalusers" config_version: 2 bitbucket: # Default "bitbucket-changeit" password which will be updated by automatic provisioning hash: "$2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC" backend_roles: - "admin" description: "Admin user"

5. Update $BITBUCKET_HOME/shared/search/config/opensearch.yml (Skip this step for Bitbucket 8).

   1 2 3 4 5 6 7 8 # vi $BITBUCKET_HOME/shared/search/config/opensearch.yml # cat $BITBUCKET_HOME/shared/search/config/opensearch.yml <existing content> plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: bitbucket.pem plugins.security.ssl.http.pemkey_filepath: bitbucket-key.pem plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem plugins.security.allow_unsafe_democertificates: false

6. Start Bitbucket.

7. Run securityadmin.sh.

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 # /bin/bash +x $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/tools/securityadmin.sh --port 7992 \ -i .bitbucket_security_index_1 \ -f $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml \ -t internalusers \ -icl \ -nhnv \ -cacert $BITBUCKET_HOME/shared/search/config/root-ca.pem \ -cert $BITBUCKET_HOME/shared/search/config/bitbucket.pem \ -key $BITBUCKET_HOME/shared/search/config/bitbucket-key.pem WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java Security Admin v7 Will connect to localhost:7992 ... done Connected as "CN=BITBUCKET" OpenSearch Version: 2.18.0 Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ... Clustername: bitbucket_search Clusterstate: YELLOW Number of nodes: 1 Number of data nodes: 1 .bitbucket_security_index_1 index already exists, so we do not need to create one. Populate config from /root Force type: internalusers Will update '/internalusers' with $$BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml    SUCC: Configuration for 'internalusers' created or updated SUCC: Expected 1 config types for node {"updated_config_types":["internalusers"],"updated_config_size":1,"message":null} is 1 (["internalusers"]) due to: null Done with success

   ℹ️ For Bitbucket 8, please use --port 7993 instead.

8. For Bitbucket 8, test and update the search password in the Server Settings page. For Bitbucket 9, please proceed to the next step.

9. Stop Bitbucket.

10. Revert the changes on the $BITBUCKET_HOME/shared/search/config/opensearch.yml.

11. Start Bitbucket.

12. Test and update the search password in the Server Settings page.

Workaround (with data loss)

1. Log in to the Bitbucket machine (SSH and any remote connection).

2. Stop Bitbucket. ( Check to see if the search process is down. You can do this by running the Linux command: ps -ef | grep java. If the search service is still running, use kill -9 <PID> to terminate it. A running search process might prevent the search directory from being regenerated after it is removed.)

3. Hash a new password by running the following command:

   1 2 3 4 # /bin/bash +x $BITBUCKET_INSTALL/opensearch/plugins/opensearch-security/tools/hash.sh WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java [Password:] <qwerty> $2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC

4. Update the hash value in $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml.

   1 2 3 4 5 6 7 8 9 10 11 # vi $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml # cat $BITBUCKET_INSTALL/opensearch/config/opensearch-security/internal_users.yml _meta: type: "internalusers" config_version: 2 bitbucket: # Default "bitbucket-changeit" password which will be updated by automatic provisioning hash: "$2y$12$xLj36E5N.Sgpymk7Ut1FvOeZYh6n9fwr.zBUbHieTA7oK12QWFdNC" backend_roles: - "admin" description: "Admin user"

5. Delete all the contents of $BITBUCKET_HOME/shared/search.

   1 # rm -rf $BITBUCKET_HOME/shared/search

6. Start Bitbucket.

7. Test and update the search password in the Server Settings page.

Unsupported platforms

For Elasticsearch

Updating the credentials on the buckler.yml and restarting the Elasticsearch service should fix this issue.

1. Shutdown Elasticsearch and Bitbucket Server

   1 # service atlbitbucket stop

2. Edit the buckler.yml file. Back it up first

   1 # vi /var/atlassian/application-data/bitbucket/shared/search/buckler/buckler.yml

3. Right after the text "auth.basic.password:" remove the current default password and replace it with any password of your choice.

4. Next, start up Bitbucket Server and ensure you're able to access Elasticsearch at http://localhost:7992 in the browser.

   1 # service atlbitbucket start

   • Perform a test and enter the credentials on the pop-up dialog with the updated credentials to see if you're able to log in.

5. If you're able to log in, attempt to test with the same credentials from the Bitbucket Server Administration area

6. If you are not able to log in:

   1. Navigate to Administration > Server Settings > Search

   2. Paste the chosen password from Step 3 into the "Search server password" field

For OpenSearch on Windows

Step 1: Create a new password hash

Passwords are stored in a hash format in a file named internal_users.yml file. As the name suggests this file stores user credentials including the admin user. Therefore, the first step is to create the hash. Opensearch provides a script to create a password hash.

1. For Windows, you have to enter into the <Bitbucket-installation-directory>\opensearch\plugins\opensearch-security\tools> and run the hash.bat file

You will be prompted to enter a password and outputs its hash. You can use bitbucket-changeit for example. Copy the output as we are going to store it in internal_users.yml

Here C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0 is our installation directory for Bitbucket 7.21.0.

Step 2: Update internal_users.yml

1. Go to <Bitbucket-installation-directory>\opensearch\plugins\opensearch-security\securityconfig

2. Open the internal_users.yml file in a text editor

3. Find the “admin” user section and update the hash value with what you got in the previous step:

Step 3: Apply security changes

1. Run the below command with your custom paths for Bitbucket home and installation directory. In the below command, the home path is D:\bitbucket-home\ and the installation directory path is C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0.

2. Open a Powershell Window or Command Prompt in the C:\Atlassian\Bitbucket\atlassian-bitbucket-7.21.0\opensearch\plugins\opensearch-security\tools and run the below command:

3. This is the output you will get:

Lastly, attempt to test with the same credentials from the Bitbucket Server Administration area under Search