Bitbucket Smart Mirrors are out of sync or will only syncronize with a delay
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Repositories that were configured to be synchronized to a Smart Mirror are not in sync with the repository on the main Data Center instance. This may be permanent (i.e. the synchronization stopped at some point and the repository never synchronizes again) or temporary (i.e. the synchronization does work but only with a delay).
The following error appears in atlassian-bitbucket.log on the primary Data Center instance:
1
2
3
2022-01-26 11:19:24,866 WARN [http-nio-7990-exec-9] @65VOI8x679x282861x6 10.10.10.15,0:0:0:0:0:0:0:1 "POST /rest/mirroring/latest/authenticate HTTP/1.0" c.a.j.i.s.DefaultAuthenticationResultHandler Failure during JWT authentication
com.atlassian.jwt.exception.JwtExpiredException: Expired at Fri Feb 25 11:15:12 CST 2022 and time is now Sat Fri 25 11:19:24 CST 2022 (30 seconds leeway is allowed)
at com.atlassian.jwt.core.reader.NimbusJwtReader.read(NimbusJwtReader.java:144)Environment
Any Bitbucket Data Center environment that has Smart Mirroring configured.
Diagnosis
There is a 7-minute difference between these systems. Requests will be rejected if the systems are not in time sync, as seen in the messages on the upstream. The large time difference is the cause mirror being unable to authenticate with the server. They need to be less than 30 seconds different!
Some commands that can be used to diagnose the NTP on all of the Bitbucket Data Center Nodes and the Bitbucket Smart Mirror:
1
ntpdate -q <ntp server host>ℹ️ This command above will not set the time. It will only test the time against the NTP server.
1
chkconfig --list ntpdThis command above shows if your ntp service is enabled, and at what runlevels.
1
service ntp statusThis command above will show if the service is running or not.
Cause
The system clock on the Smart Mirror(s) and the primary Data Center instance are not synchronized and are more than 30 seconds apart.
Note: It is acceptable for the primary Data Center instance and the Smart Mirror nodes to be in different time zones.
Solution
Resolution
Set up synchronization with a Network Time Protocol (NTP) server on all involved machines, i.e. all Data Center nodes and all Smart Mirror nodes.
Was this helpful?