Memberships from large user groups are removed after a directory sync in Jira when using a load balanced LDAP
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When a load balanced Active Directory / LDAP directory service is configured for a directory, group memberships are removed and added intermittently after a directory syncronization when the group has a large amount of members.
A subset of users are removed in some syncs, only to be added again by other syncs. There is no change of the group memberships on the directory service.
Environment
Jira Server or Data Center
Jira is connected to a LDAP load balancer, that is, an intermediary service sitting between a domain controller or LDAP server and Jira, that directs LDAP traffic to different AD/LDAP servers
Diagnosis
Your environment matches above, and
The group affected is has a membership count greater than the LDAP/AD server maximum (default of 1500 in Active Directory), and
When viewing the Audit Log for a sample user, the user is removed from the directory, then added again on a future sync (IE: flapping), and
When replicating the same group membership query from the Jira host operating system using a tool like
ldapsearch
, you see an inconsistent result returned between ranges. For example, range 1 contains a sample user, but range 2 also contains the sample user. To check:Obtain the response from the LDAP load balancer for both the problematic sample user, and the problematic group, checking 20 times.
ldapsearch_test.sh
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
#!/bin/bash ## ## Perform a user lookup via ldapsearch, ranging the results, and storing to dated text files ## (i) Configure the placeholders below, including the ranges according to your LDAP server requirements. Adjust the number of queries and ranges based on the membership count for the target group ## In the sample below, we use a range size of 1500, and expect to see up to 6000 members for i in $(seq 20); do now=`date +%s`; ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "DC=your,DC=base,DC=dn,DC=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_USER_FILTER_HERE)" memberOf > $now.ldapuser.txt; ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=0-1499' > $now.ldapgroup_0.txt; ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=1500-2999' > $now.ldapgroup_1.txt; ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=3000-4499' > $now.ldapgroup_2.txt; ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=4500-6000' > $now.ldapgroup_3.txt; sleep 20m; done;
Execute the following python script against the directory containing the above files, for example:
python3 parse.py "/path/to/directory/"
parse.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52
import argparse import os.path from os import listdir from os.path import isfile, join def member_line(line: str): return line.startswith('member;range') parser = argparse.ArgumentParser(description='Find LDAP duplicates') parser.add_argument('path', metavar='path', type=str, help='path to files') args = parser.parse_args() filegroups = {} onlyfiles = sorted([f for f in listdir(args.path) if isfile(join(args.path, f))]) for filename in onlyfiles: split_filename = filename.split('.') if 'ldapgroup' in split_filename[1]: groupname = split_filename[0] if groupname not in filegroups: filegroups[groupname] = [] filegroups[groupname].append(filename) for groupname, filenames in filegroups.items(): print(f'Processing {groupname}') occurrences_group = {} total_group = 0 for filename in filenames: with open(args.path + filename) as f: lines = f.read().splitlines() occurrences_file = {} for i, line in enumerate(lines): if member_line(line): dn = line.split(":")[1] if i + 1 < len(lines) and not member_line(lines[i+1]): dn += lines[i+1].strip() total_group += 1 if dn not in occurrences_file: occurrences_file[dn] = 0 occurrences_file[dn] += 1 for key, value in occurrences_file.items(): if key not in occurrences_group: occurrences_group[key] = 0 occurrences_group[key] += value duplicates_group = {k: v for k, v in occurrences_group.items() if v > 1} if len(duplicates_group) > 0: # for duplicate, count in duplicates_group.items(): # print(f'Duplicated user {duplicate}, occurrences {count}') print(f'Found duplicates in group {groupname}, sample user: {list(duplicates_group.keys())[0]}') print(f'Total: {total_group}, Unique member entries: {len(occurrences_group)}')
Check the script's output. In each run, the "Total" should match exactly the "unique member entries."
Example healthy output: (meaning this KB does not apply)
1 2 3 4 5 6 7 8
Processing 1632103058 Total: 6233, Unique member entries: 6233 Processing 1632103958 Total: 6233, Unique member entries: 6233 Processing 1632104859 Total: 6234, Unique member entries: 6234 Processing 1632105760 Total: 6234, Unique member entries: 6234 ...
Example bad output: (meaning this KB applies)
1 2 3 4 5 6 7 8 9
Processing 1632085541 Found duplicates in group 1632085541, sample user: CN=Test\, user,OU=example,DC=example,DC=com Total: 6234, Unique member entries: 6110 Processing 1632086742 Found duplicates in group 1632086742, sample user: CN=User\, Name,OU=example,DC=example,DC=com Total: 6234, Unique member entries: 6116 Processing 1632087943 Found duplicates in group 1632087943, sample user: CN=Test\, user,OU=example,DC=example,DC=com Total: 6234, Unique member entries: 6221 ...
Cause
During the directory syncronization process, a query to fetch the membership of each group is issued to the LDAP server defined in the directory configuration.
If the membership count on the LDAP side of any group is larger than a certain value (in AD, default 1500), only those memberships up to that count are returned by the server, and, Jira must issue a second LDAP search command with an adjusted range to the server. This process repeats until there are no more ranges to fetch.
This process works well in normal situations, and, is an expected part of any LDAP client speaking to a LDAP server. However, if a load balancer in between Jira interferes with this, for example, routing the first query to one server, then the second to a different server, if these two servers have slightly different contents, the range could be offset in a way that duplicates or all together removes results.
This does not affect a load-balancer-less normal ActiveDirectory DNS round robin scenario, where the server name is the simply the domain itself, eg, corp.example.com, as the operating system will resolve the name to an IP, cache it, then return subsequent requests to the same server. The problem is introduced in some load balancers than do not forward subequent queries to the same server.
Solution
Modify the directory within Jira so that it only connects to a single server, eg, a single domain controller, or, the FQDN of the domain, then run a directory full synchronization within Jira or
Modify the load balancer configuration to forward subsequent ranged requests to the same server
Other notes
Microsoft Documentation on range retrieval, including limits in Windows Server Active Directory: https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ldap/searching-using-range-retrieval
Was this helpful?