After upgrading Crowd Data Center, directory synchronisations do not complete for some nodes, with "stalled synchronisations" entries in the logs.
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After upgrading the Crowd Data Center, the directory synchronization never ends. The lock on the specific directory from a cluster node has not been released.
Environment
5.1.3
Diagnosis
In the application logs of Crowd, we can see the below lines regarding the synchronization of the specific directory ID;
1
2023-08-14 15:25:00,011 Caesium-2-4 INFO [crowd.manager.directory.FailedSynchronisationManagerImpl] Found 1 stalled synchronisations for directories [ [<Directory ID>] ]. Rescheduling them to run againIf the NODE_ID shows us one of the cluster nodes, it means that the cluster didn't release the lock on this specific directory.
The below SQL query helps us to identify which node is locking the specific directory;
ℹ️ Please change the <Directory ID> with the identified one from the log file before executing the query.
1
SELECT * FROM cwd_cluster_lock WHERE LOCK_NAME = 'com.atlassian.crowd.embedded.api.Directory:<Directory ID>';Cause
We haven't identified the cause of this specific situation yet. When we have clear reproduction steps, we will update the article accordingly.
It may be that this issue can occur if Crowd was forcefully shut down at some point.
Solution
If one of the cluster nodes locks the directory, the below steps can help us release the lock on it.
Stop Crowd
⚠️Take a full backup of the database to prevent any data loss.
Execute the below query;
1UPDATE CWD_CLUSTER_LOCK SET NODE_ID=NULL WHERE LOCK_NAME = 'com.atlassian.crowd.embedded.api.Directory:<Directory ID>';Start Crowd
Then, try to resync the affected directory.
Was this helpful?