Bitbucket Mirror unable to sync all repositories
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Some repositories are unable to sync between the Mirror and Upstream server.
Diagnosis
The Mirror nodes page on the Bitbucket UI would show the status as Synchronizing and be stuck there.
The Mirror node will contain the following logs:
1
2
3
4
5
6
ERROR [threadpool:thread-1] c.a.b.i.m.m.s.FullSynchronizationHelper [<Bitbucket Base URL>]#code: full synchronization failed
com.atlassian.bitbucket.internal.mirroring.mirror.UpstreamRequestFailedException: A request to the configured upstream server failed (Reason: 'org.apache.http.ConnectionClosedException: Connection closed')
... 1 frame trimmed
Caused by: org.apache.http.ConnectionClosedException: Connection closed
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:350)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)This means that the mirror instance is unable to reach the primary server. In that case, check if there are any network issues between the mirror node and your upstream server, verify the network connectivity between the two.
For diagnosing mirror issues, it's a good idea to add these properties to enable debug logging and then generate support zip:
1
2
3
4
5
6
7
logging.logger.ROOT=INFO
logging.logger.com.atlassian.bitbucket=DEBUG
logging.logger.com.atlassian.event.remote=DEBUG
logging.logger.com.atlassian.stash=DEBUG
logging.logger.org.hibernate=INFO
logging.logger.org.springframework=INFO
logging.logger.com.hazelcast.internal.diagnostics.HealthMonitor=DEBUGIt is possible that this issue occurs for only a few repositories where they are unable to sync between upstream node and the Mirror. When you clone from Mirror URL and push back, those changes would get synced with the upstream node.
After that if you push some changes to the upstream URL and then try to clone the repo again using the Mirror URL, you don't get the latest changes in the locally cloned copy which you pushed using the upstream URL.
So mirror to upstream sync would happen but upstream to mirror sync would not happen. Usually as per the normal scenario a repository gets synced at both sides whenever there are changes pushed to it from any of the URL.
From the logs, you would notice that a lot of repos have synced successfully:
1
2
3
4
INFO [initial-sync-queue-poller:thread-1] c.a.b.i.m.m.f.s.InitialSyncHelper Initial Synchronization for [6832] complete
INFO [initial-sync-queue-poller:thread-1] c.a.b.i.m.m.f.s.InitialSyncHelper Initial Synchronization for [6829] complete
INFO [initial-sync-queue-poller:thread-1] c.a.b.i.m.m.f.s.InitialSyncHelper Initial Synchronization for [10430] complete
INFO [initial-sync-queue-poller:thread-1] c.a.b.i.m.m.f.s.InitialSyncHelper Initial Synchronization for [10440] completeWe need to figure out which one of these are not getting synced. When there are multiple restarts, it's quite possible that different repo must've failed to sync each time.
Cause
The cause of all the repos not getting synced could be multiple restarts, due to which all the repositories are unable to get completely synced between the mirror and the upstream node.
From the atlassian-bitbucket.log, you might see that the indexing is running and when checking for repositories that are marked as "full synced" when you see a message similar to the following:
1
DEBUG [AtlassianEvent::thread-13] @UG7QGNx60x13936x0 c.a.b.i.m.m.DefaultUpstreamService Repository synchronized! <PROJECT-KEY>/<repo-slug>[<repo ID in mirror node>]If you see any intermittent connection closed logs, such as
Caused by: org.apache.http.ConnectionClosedException: Connection closed, there could be something outside of Bitbucket that is forcibly closing the connection between the mirror and upstream server that needs to be looked into.If you have specific repositories that are not synced after you stop seeing "completed full sync" in the logs, such as
Repository: [<PROJECT-KEY>/<repo-slug>]#10444 completed full sync, starting snapshot sync, check on those repositories directly in the Bitbucket logs.Otherwise, if the mirror is being restarted in the middle of confirming its sync status with the upstream, it causes further delays.
Solution
Let the mirror continue to run and allow the initial sync to complete and then complete the full syncs that are needed.
Was this helpful?