Bitbucket nodes do not form a cluster if the nodes have multiple network interfaces and Multicast discovery is used
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket Data Center nodes may not form a cluster if one or more nodes has multiple network interfaces and IP addresses and if multicast is used for node discovery.
Bitbucket listens on all network interfaces by default. It does not choose the specific network interface to use and relies on the underlying Operating System to assign it an interface.
When a node has more than one IP address, the "wrong" IP address may get selected which would lead to the nodes not discovering each other.
Environment
Bitbucket Data Center instances with:
more than one node
one or more nodes have multiple network interfaces and IP addresses
using multicast as node discovery mechanism,
i.e. the following property is set in the <
BITBUCKET_HOME>/shared/bitbucket.propertiesfile:1 2# Use multicast to discover cluster nodes hazelcast.network.multicast=true
Diagnosis
Enable debug logging for Bitbucket.
When a Data Center cluster is being formed, the $BITBUCKET_HOME/log/atlassian-bitbucket.log will have logs that indicate that a new node was authenticated and added to the cluster.
Sample logs
These are sample logs for the existing node:
1
2
3
4
5
6
7
8
9
10
11
12
13
2023-04-21 08:30:25,654 DEBUG [hz.hazelcast.cached.thread-3] c.a.s.i.c.DefaultClusterJoinManager ACCEPT(10.255.9.1:5701 <- 10.255.9.3:38113): Authenticating cluster node
2023-04-21 08:30:25,865 DEBUG [hz.hazelcast.cached.thread-3] c.a.s.i.c.SharedSecretClusterAuthenticator Created: Response{proof=C1BC540D6526815590745657F64488C68051FD85ED786CE79BA53286A7F7A1B5}
2023-04-21 08:30:26,306 DEBUG [hz.hazelcast.cached.thread-3] c.a.s.i.c.DefaultClusterJoinManager ACCEPT(10.255.9.1:5701 <- 10.255.9.3:38113): Node authenticated successfully
2023-04-21 08:30:26,557 INFO [hz.hazelcast.IO.thread-in-0] c.hazelcast.nio.tcp.TcpIpConnection [10.255.9.1]:5701 [bitbucket-docker] [3.12.13] Initialized new cluster connection between /10.255.9.1:5701 and /10.255.9.3:38113
2023-04-21 08:30:31,643 INFO [hz.hazelcast.event-1] c.a.s.i.c.HazelcastClusterService Node '/10.255.9.3:5701' was ADDED to the cluster. Updated cluster:
[/10.255.9.1:5701 master this uuid='31eeeedf-3c2c-4a37-947d-a983ad4399d4' vm-id='44ff2601-eb98-46b6-9578-e1e804c5a6ab'],
[/10.255.9.3:5701 uuid='3e10e740-313b-4e0f-974e-686a05785240' vm-id='457c6c42-b8f7-49b7-a194-54a549f48a1f']
2023-04-21 08:30:31,644 INFO [hz.hazelcast.priority-generic-operation.thread-0] c.h.internal.cluster.ClusterService [10.255.9.1]:5701 [bitbucket-docker] [3.12.13]
Members {size:2, ver:3} [
Member [10.255.9.1]:5701 - 31eeeedf-3c2c-4a37-947d-a983ad4399d4 this
Member [10.255.9.3]:5701 - 3e10e740-313b-4e0f-974e-686a05785240 lite
]These are sample logs on the new node that recently joined the cluster:
1
2
3
4
5
6
7
8
2023-04-21 08:31:29,725 INFO [spring-startup] c.a.s.i.hazelcast.HazelcastLifecycle Promoting Hazelcast instance at 10.255.9.3:5701 to full member
2023-04-21 08:31:29,728 INFO [hz.hazelcast.priority-generic-operation.thread-0] c.h.i.cluster.impl.MembershipManager [10.255.9.3]:5701 [bitbucket-docker] [3.12.13] Member [10.255.9.3]:5701 - 3e10e740-313b-4e0f-974e-686a05785240 this lite is promoted to normal member.
2023-04-21 08:31:29,729 INFO [hz.hazelcast.priority-generic-operation.thread-0] c.h.internal.cluster.ClusterService [10.255.9.3]:5701 [bitbucket-docker] [3.12.13]
Members {size:2, ver:4} [
Member [10.255.9.1]:5701 - 31eeeedf-3c2c-4a37-947d-a983ad4399d4
Member [10.255.9.3]:5701 - 3e10e740-313b-4e0f-974e-686a05785240 this
]If similar logs are not found, the nodes are not discovering each other.
Solution
Option 1
Use TCP/IP instead of Multicast for node discovery.
Update the <
BITBUCKET_HOME>/shared/bitbucket.propertiesfile and set the following:1 2 3 4 5 6 7 8 9# Disable multicast hazelcast.network.multicast=false # Use TCP/IP hazelcast.network.tcpip=true # IP addresses of some or all of the cluster nodes. # Not all of the cluster nodes have to be listed here but at least one of them has to be active when a new node joins hazelcast.network.tcpip.members=192.168.0.1:5701,192.168.0.2:5701,192.168.0.3:5701
Option 2
If multicast is still preferred, the following workaround can be used, which involves creating a new bitbucket.properties file that's local to the node
Create a new
bitbucket.propertiesfile located at<BITBUCKET_HOME>. This file is not shared and is local to the specific nodeAdd the following property to the local
<BITBUCKET_HOME>/bitbucket.propertiesfile:1 2# Specify the public address to be used explicitly hazelcast.local.public.address=192.168.0.1:5701
Was this helpful?