Bitbucket DC nodes in AWS do not form the cluster

Summary

Bitbucket DC nodes in AWS cannot form the cluster.

• The first node starts up correctly and is fully functional.

• The other nodes also start up, but they cannot join the cluster that was previously created by the first node.

Environment

• Bitbucket DC running in AWS

• Hazelcast has AWS EC2 Auto Discovery enabled and configured to use AWS IAM role

• Product version:

  This issue has been confirmed with Bitbucket DC version 8.9.22, but it may also occur on other versions.

Diagnosis

• Each node logs the following messages during the startup sequence:

  2025-03-27 07:15:56,436 INFO [main] c.a.b.i.b.BitbucketServerApplication Starting BitbucketServerApplication v8.9.22 using Java 11.0.8 on ip-123-45-67-89.eu-central-1.compute.internal with PID 22870 (/app/atlassian/install/atlassian-bitbucket-8.9.22/app/WEB-INF/classes started by bitbucket in /) 2025-03-27 07:15:56,437 INFO [main] c.a.b.i.b.BitbucketServerApplication No active profile set, falling back to 1 default profile: "default" 2025-03-27 07:15:57,159 INFO [main] c.a.b.i.boot.log.BuildInfoLogger Starting Bitbucket 8.9.22 (dd450d9 built on Sun Nov 17 23:43:20 UTC 2024) 2025-03-27 07:15:57,159 INFO [main] c.a.b.i.boot.log.BuildInfoLogger JVM: AdoptOpenJDK OpenJDK 64-Bit Server VM 11.0.8+10 2025-03-27 07:15:58,419 INFO [main] c.a.b.i.b.BitbucketServerApplication Started BitbucketServerApplication in 3.027 seconds (JVM running for 4.0) 2025-03-27 07:16:01,419 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /app/atlassian/home/bitbucket 2025-03-27 07:16:04,884 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Successfully acquired lock on home directory /app/atlassian/home/bitbucket/shared 2025-03-27 07:16:06,549 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [1] retrying in 1 seconds... 2025-03-27 07:16:08,050 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [2] retrying in 2 seconds... 2025-03-27 07:16:10,302 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [3] retrying in 3 seconds... 2025-03-27 07:16:13,678 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [4] retrying in 5 seconds... 2025-03-27 07:16:18,741 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [5] retrying in 7 seconds... 2025-03-27 07:16:26,336 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [6] retrying in 11 seconds... 2025-03-27 07:16:37,726 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [7] retrying in 17 seconds... 2025-03-27 07:16:54,810 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [8] retrying in 25 seconds... 2025-03-27 07:17:20,436 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [9] retrying in 38 seconds... 2025-03-27 07:17:58,873 WARN [spring-startup] com.hazelcast.aws.utility.RetryUtils Couldn't connect to the AWS service, [10] retrying in 57 seconds... 2025-03-27 07:18:56,533 WARN [spring-startup] c.hazelcast.aws.AwsDiscoveryStrategy Cannot discover nodes, returning empty list com.hazelcast.config.InvalidConfigurationException: Unable to retrieve credentials from IAM Role: <IAM_role> at com.hazelcast.aws.impl.DescribeInstances.fillKeysFromIamRole(DescribeInstances.java:134) at com.hazelcast.aws.impl.DescribeInstances.fillKeysFromIamRoles(DescribeInstances.java:114) at com.hazelcast.aws.impl.DescribeInstances.execute(DescribeInstances.java:254) at com.hazelcast.aws.AWSClient.getAddresses(AWSClient.java:57) at com.hazelcast.aws.AwsDiscoveryStrategy.discoverNodes(AwsDiscoveryStrategy.java:146) at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverNodes(DefaultDiscoveryService.java:71) at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddresses(DiscoveryJoiner.java:70) at com.hazelcast.internal.cluster.impl.DiscoveryJoiner.getPossibleAddressesForInitialJoin(DiscoveryJoiner.java:59) at com.hazelcast.cluster.impl.TcpIpJoiner.joinViaPossibleMembers(TcpIpJoiner.java:151) at com.hazelcast.cluster.impl.TcpIpJoiner.doJoin(TcpIpJoiner.java:111) at com.hazelcast.internal.cluster.impl.AbstractJoiner.join(AbstractJoiner.java:137) at com.hazelcast.instance.Node.join(Node.java:820) at com.hazelcast.instance.Node.start(Node.java:455) at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136) at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203) at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182) at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132) at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57) at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.newInstance(HazelcastFactoryBean.java:126) at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:66) at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:34) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591) at javax.servlet.GenericServlet.init(GenericServlet.java:143) at java.base/java.lang.Thread.run(Thread.java:834) ... 34 frames trimmed Caused by: com.hazelcast.config.InvalidConfigurationException: Unable to lookup role in URI: http://169.254.169.254/latest/meta-data/iam/security-credentials/<IAM_role> at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:78) at com.hazelcast.aws.utility.MetadataUtil$1.call(MetadataUtil.java:109) at com.hazelcast.aws.utility.MetadataUtil$1.call(MetadataUtil.java:106) at com.hazelcast.aws.utility.RetryUtils.retry(RetryUtils.java:52) at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:106) at com.hazelcast.aws.impl.DescribeInstances.retrieveRoleFromURI(DescribeInstances.java:170) at com.hazelcast.aws.impl.DescribeInstances.fillKeysFromIamRole(DescribeInstances.java:131) ... 25 common frames omitted Caused by: java.io.FileNotFoundException: http://169.254.169.254/latest/meta-data/iam/security-credentials/<IAM_role> at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1920) at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520) at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:70) ... 31 common frames omitted

• Bitbucket's bitbucket.properties file contains the following two configuration lines, where the latter references the same <IAM_role> as seen in the warning quoted above:

  hazelcast.network.aws=true hazelcast.network.aws.iam.role=<IAM_role>

• The following curl command returns a 404 error:

• curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<IAM_role>

Cause

• The IAM role configured in bitbucket.properties is different from the one expected by AWS.

  The IAM role may have changed in AWS (for example, after an OS upgrade), but bitbucket.properties may still have its old value.

Solution

1. Check the correct value of IAM role in AWS. The following curl command, executed on one of the nodes, should return the correct IAM role:

   curl http://169.254.169.254/latest/meta-data/iam/security-credentials/

2. Run the following curl command to validate the correct IAM role:

   curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<New_IAM_role>

   The <New_IAM_role> needs to be replaced with the value returned in the first step.

   • If the <New_IAM_role> is correct, you should receive a valid, non-404 response, with "Code" : "Success" and "Token" : "<The_Value_Of_The_Token>"

3. Edit Bitbucket's bitbucket.properties file (located in $BITBUCKET_HOME/shared) and replace the value that follows hazelcast.network.aws.iam.role= with the one identified in the first step.

4. Shut down all Bitbucket nodes, then start just one node.

5. Once the web interface is responsive and functional, start another node.

6. Observe in the web interface (under Administration > Clustering) that the second node joined the cluster.

7. Start the remaining nodes, one by one, always making sure that the last node has joined the cluster before the next one is started.