Health Check in Jira DataCenter reports warning with Cluster Index Replication followed by critical message with Cluster Time Synchronisation
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The health check shows the following warning message for the cluster index replication:
1
Index replication for cluster node 'node2' is behind by 121 seconds.
After a few days, we see the health check reporting the following critical message for the Cluster Time Synchronization:
1
System clocks of nodes are not synchronized. Difference: 46990ms. Offsets: node1:-6684ms node2:-53647ms
Environment
Jira Data Center on any version from 9.0.0.
Diagnosis
Follow the steps below to troubleshoot the issue further when the Health Check warning message is seen:
Check the size of the index queue on each Jira node by following the steps below:
Open the URL http://<Jira-Node-URL>/rest/api/2/index/summary
Wait for 10 minutes
Run the same endpoint again on all nodes separately
Verify that the queue size is 0 and the count of issues in index and Database is consistent.
Create a few issues on one node to see if it replicates on the other node, by following the steps below:
Create a dummy project on node 2.
Create 6-7 issues in the project.
Check on node 1 if it replicates fine.
Do the same from the node 1 and check on the node 2 this time.
Verify that the issues are visible in the other nodes immediately, which indicates that index replication is working as expected
Look at the timezone of the server and the database, and verify that they are in the same timezone: the Jira JVM is in sync with the node and DB timezone
Run the following command in the servers hosting Jira and the database, in order to look closely at the server and the database timezones
1timedatectlIf running the above command show a difference in the time by a minute between the servers and the database even though they are in the same timezone, then this KB article is relevant
Root Cause
The NTP servers were not configured and since the time was not in sync, the health check reported the synchronization errors.
Solution
Refer to the following article and make sure to either configure the NTP server or manually set the time of nodes to be in-sync.
Was this helpful?