LDAP Directory Syncing indefinitely

Summary

The synchronization process from one (or more) of your LDAP connections synchronizing for way longer than usual (Example: "Synchronising for 43158s..."), and seems to be stuck.

Environment

• Jira Data Center.

Diagnosis

To make sure that the synchronization process has really become stuck or if it is just processing a lot of information (possibly due to changes that occurred on the LDAP side), we'll need to follow the steps below:

1. Begin by running the following select query on the Jira DB, which will provide us with the ID of the directory that's syncing along with which node owns the synchronization process.

   1 SELECT * FROM cwd_synchronisation_status;

2. Now create a new support zip with thread dumps (you'll likely need to manually adjust the support zip customization to include that) from the 'node_id' returned above.

3. Access the atlassian-jira.log, from either the Jira Home directory or the newly created support zip, and look for the most recent occurrence of the LDAP synchronization for the 'directory_id' you received on step 1 (in this example the directory_id is '10200').

   1 2022-03-18 02:26:00,484+0100 Caesium-1-1 INFO anonymous [c.a.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation for directory [ 10200 ] starting

4. With the above information, we can confirm what is the name of the thread responsible for this sync occurrence (In our example it was 'Caesium-1-1').

5. Now look for this thread name in the '<Support_zip>/thread-dump/threaddump_xxxxx.tdump' file.

6. Confirm that the occurrences of this thread are NOT in a RUNNING state, or if they are running, that its stack traces make no references to 'com.atlassian.crowd' classes.

If the 'sync_status' column from step 1 is not 'SUCCESS_FULL', and the threads from step 6 are not RUNNING, or at least not making references to 'com.atlassian.crowd' classes, the synchronization process has likely become stale, and so, you may proceed with the resolution steps below.

Cause

Sometimes, the LDAP synchronization process can become orphaned before it has the opportunity to update either the application cache or the database record about the synchronization. 

A few causes for this are an ungraceful Jira shutdown during a synchronization process, or the occurrence of a StackOverflow during the LDAP sync.

When this happens, Jira believes that the synchronization process is still running; however, its thread has long abandoned it.

Solution

In order to resolve the problem with the stuck synchronization process, we'll need to restart the entire Jira cluster (which will have the triple benefit of allowing us to safely manipulate the database, close any Jira thread and clear flush the entire cluster cache), along with manually removing the stale synchronization record form the DB.

1. Shut down Jira on all nodes.

2. Backup the Jira database using DB native tools.

3. Run the following SQL statement:

   1 DELETE FROM cwd_synchronisation_status;

4. Start Jira again, one node at a time.