Bitbucket Data Center node fails to join cluster - Current node is unable to service requests
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Your newly provisioned Bitbucket Server node fails to join the cluster:
atlassian-bitbucket.log
2016-11-07 11:33:41,179 WARN [hz.hazelcast.IO.thread-Acceptor] com.hazelcast.nio.tcp.SocketAcceptor [10.137.158.47]:5701 [testbitbucket] [3.5.2-atlassian-37] com.atlassian.stash.internal.cluster.NodeConnectionException: Required property 'sharedHome' should be '/storage/data/bitbucket/shared' but is '/storage/data/bitbuckethome/shared', Required property 'timeZone' should be 'UTC' but is 'Australia/Victoria'
com.atlassian.stash.internal.cluster.NodeConnectionException: Required property 'sharedHome' should be '/storage/data/bitbucket/shared' but is '/storage/data/bitbuckethome/shared', Required property 'timeZone' should be 'UTC' but is 'Australia/Victoria'
at com.atlassian.stash.internal.cluster.DefaultClusterJoinManager.negotiateOutcome(DefaultClusterJoinManager.java:254) ~[bitbucket-service-impl-4.10.1.jar:na]
at com.atlassian.stash.internal.cluster.DefaultClusterJoinManager.accept(DefaultClusterJoinManager.java:99) ~[bitbucket-service-impl-4.10.1.jar:na]
at com.atlassian.stash.internal.hazelcast.ClusterJoinSocketInterceptor.onAccept(ClusterJoinSocketInterceptor.java:44) ~[bitbucket-service-impl-4.10.1.jar:na]
at com.hazelcast.nio.tcp.TcpIpConnectionManager.interceptSocket(TcpIpConnectionManager.java:165) ~[hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
at com.hazelcast.nio.tcp.SocketAcceptor.configureAndAssignSocket(SocketAcceptor.java:142) [hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
at com.hazelcast.nio.tcp.SocketAcceptor.acceptSocket(SocketAcceptor.java:128) [hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
at com.hazelcast.nio.tcp.SocketAcceptor.run(SocketAcceptor.java:70) [hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
... 20 frames trimmedCause
When joining the cluster, a set of parameters need to be the same between the node. That's a requirement.
In the stack trace above, the shared directory and the TimeZone diverge between nodes.
The timezone of the affected node does not match that of the cluster:
atlassian-bitbucket.log
2015-05-07 17:59:42,508 DEBUG [hz.hazelcast.cached.thread-1] c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Starting to connect to Address[192.168.1.3]:5701
2015-05-07 17:59:42,509 DEBUG [hz.hazelcast.cached.thread-1] c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Connecting to /192.168.1.3:5701, timeout: 0, bind-any: true
2015-05-07 17:59:42,509 DEBUG [hz.hazelcast.cached.thread-1] c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Successfully connected to: Address[192.168.1.3]:5701 using socket Socket[addr=/192.168.1.3,port=5701,localport=52601]
2015-05-07 17:59:42,533 DEBUG [hz.hazelcast.cached.thread-1] c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Could not connect to: /192.168.1.3:5701. Reason: NodeConnectionException[ 'sun.util.calendar.ZoneInfo[id="US/Pacific-New",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific-New,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]' but is 'sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]']
2015-05-07 17:59:42,534 DEBUG [hz.hazelcast.cached.thread-1] c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Required property 'timeZone' should be 'sun.util.calendar.ZoneInfo[id="US/Pacific-New",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific-New,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]' but is 'sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]'
com.atlassian.bitbucket.internal.cluster.NodeConnectionException: Required property 'timeZone' should be 'sun.util.calendar.ZoneInfo[id="US/Pacific-New",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific-New,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]' but is 'sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]'
at com.atlassian.bitbucket.internal.cluster.DefaultClusterJoinManager.checkNoRequirementErrors(DefaultClusterJoinManager.java:284) ~[DefaultClusterJoinManager.class:na]
at com.atlassian.bitbucket.internal.cluster.DefaultClusterJoinManager.checkRequirementsMatchServer(DefaultClusterJoinManager.java:213) ~[DefaultClusterJoinManager.class:na]
at com.atlassian.bitbucket.internal.cluster.DefaultClusterJoinManager.joinServer(DefaultClusterJoinManager.java:81) ~[DefaultClusterJoinManager.class:na]
at com.atlassian.bitbucket.internal.hazelcast.ClusterJoinSocketInterceptor.onConnect(ClusterJoinSocketInterceptor.java:48) ~[ClusterJoinSocketInterceptor.class:na]
at com.hazelcast.nio.tcp.TcpIpConnectionManager.interceptSocket(TcpIpConnectionManager.java:155) ~[TcpIpConnectionManager.class:3.3-RC3]
at com.hazelcast.nio.tcp.SocketConnector.tryToConnect(SocketConnector.java:131) ~[SocketConnector.class:3.3-RC3]
at com.hazelcast.nio.tcp.SocketConnector.run(SocketConnector.java:67) ~[SocketConnector.class:3.3-RC3]
at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) [CachedExecutorServiceDelegate$Worker.class:3.3-RC3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09-icedtea]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]
at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) [PoolExecutorThreadFactory$ManagedThread.class:3.3-RC3]
... 1 frame trimmedResolution
Update the node's time zone to match that of the cluster by symlinking /etc/localtime to the correct time zone file usually found here: /usr/share/zoneinfo
How to identify there is still a JVM mismatch even after the symlink above?
If the node's time zone matches the other nodes in the cluster but Bitbucket Server still reports a mismatch, you have force the JVM to acknowledge the timezone you want by passing an additional parameter to the JVM (Java Virtual Machine).
How to identify the cause for the persistent mismatch?
You can checkout the repository below and run the
jarprogram to output the Time Zone on each one of your nodes. The code can be found here. Use the procedure below:Checkout the code into your node:
$ git clone git@bitbucket.org:thiago_bomfim/gettimezone.git $ cd gettimezone/targetList the process running Bitbucket server:
ps -ef | grep bitbucket | grep -v elastic atlbitb+ 843 1 0 Aug23 ? 00:14:40 /usr/java/jre1.8.0_73/bin/java -Djava.util.logging.config.file=/opt/atlassian/bitbucket/4.7.1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx768m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Datlassian.standalone=BITBUCKET -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.apache.catalina.connector.Response.ENFORCE_ENCODING_IN_GET_WRITER=false -Djava.library.path=/opt/atlassian/bitbucket/4.7.1/lib/native:/var/atlassian/application-data/bitbucket/lib/native -Dbitbucket.home=/var/atlassian/application-data/bitbucket -Djava.endorsed.dirs=/opt/atlassian/bitbucket/4.7.1/endorsed -classpath /opt/atlassian/bitbucket/4.7.1/bin/bitbucket-bootstrap.jar:/opt/atlassian/bitbucket/4.7.1/bin/bootstrap.jar:/opt/atlassian/bitbucket/4.7.1/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/bitbucket/4.7.1 -Dcatalina.home=/opt/atlassian/bitbucket/4.7.1 -Djava.io.tmpdir=/opt/atlassian/bitbucket/4.7.1/temp com.atlassian.stash.internal.catalina.startup.Bootstrap startUse the Java running this process to output the timezone of your node. From the example above:
/usr/java/jre1.8.0_73/bin/java -jar gettimezone-1.0-SNAPSHOT.jar The Timezone of this JVM is: "Australia/Sydney"Alternatively to the method above, if you generate a Support Zip for each one of the nodes in the cluster (Administration -> Atlassian Support Tools -> Support Zip -> Create) and look into the section below, chances are you will see the mismatch:
Node 1
<java-runtime-environment>
...
<user.timezone>America/Montreal</user.timezone>
...
</java-runtime-environment>Node 2
<java-runtime-environment>
...
<user.timezone>America/Chicago</user.timezone>
...
</java-runtime-environment>How to force the JVM with the expected TZ you used in the symlink?
The following example sets the time zone for Chicago. You have to do that for each one of the nodes:
Edit
<BITBUCKET_INSTALL>/bin/setenv.sh:
<BITBUCKET_INSTALL>/bin/setenv.sh
-Duser.timezone=America/Chicagosetenv and environment variable changes in Bitbucket Server 5.0+
Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.
Linux
When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh (or start-bitbucket.sh). These values will be read when the service starts.
As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:
Example
JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryErrorWindows
Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.
Note that after applying the configuration above, a restart on your nodes is required so the node will pick up the JVM argument (i.e. after the restart you should see this parameter on the output of ps -ef|grep bitbucket)
The List of TZ Database Time Zones documentation on Wikipedia contains a comprehensive list of possible time zones that can be set.
Examples of common time zones
Timezone | Code |
|---|---|
Sydney | Australia/Sydney |
Los Angeles | America/Los_Angeles |
New York | America/New_York |
Paris | Europe/Paris |
Rome | Europe/Rome |
UTC | UTC |
Madrid | Europe/Madrid |
How to change BITBUCKET_HOME?
Your installer would have set that in these 2 different files (depending on your operating system):
<Bitbucket_Install>/bin/set-bitbucket-home.sh or
<Bitbucket_Install>/bin/set-bitbucket-home.bat
For Linux, <Bitbucket_Install> is set to /opt/atlassian/bitbucket/4.10.1/ by default.
Was this helpful?