要約

When a load balanced Active Directory / LDAP directory service is configured for a directory, group memberships are removed and added intermittently after a directory syncronization when the group has a large amount of members.

A subset of users are removed in some syncs, only to be added again by other syncs. There is no change of the group memberships on the directory service.

環境

• Jira Server または Data Center
• Jira is connected to a LDAP load balancer, that is, an intermediary service sitting between a domain controller or LDAP server and Jira, that directs LDAP traffic to different AD/LDAP servers

診断

• Your environment matches above, and
• The group affected is has a membership count greater than the LDAP/AD server maximum (default of 1500 in Active Directory), and
• When viewing the Audit Log for a sample user, the user is removed from the directory, then added again on a future sync (IE: flapping), and
• When replicating the same group membership query from the Jira host operating system using a tool like ldapsearch , you see an inconsistent result returned between ranges. For example, range 1 contains a sample user, but range 2 also contains the sample user. To check:

  1. Obtain the response from the LDAP load balancer for both the problematic sample user, and the problematic group, checking 20 times.

     ldapsearch_test.sh

     #!/bin/bash
     
     ##
     ## Perform a user lookup via ldapsearch, ranging the results, and storing to dated text files
     ## (i) Configure the placeholders below, including the ranges according to your LDAP server requirements. Adjust the number of queries and ranges based on the membership count for the target group
     ##     In the sample below, we use a range size of 1500, and expect to see up to 6000 members
     
     for i in $(seq 20);
     do
       now=`date +%s`;
       ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "DC=your,DC=base,DC=dn,DC=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_USER_FILTER_HERE)" memberOf > $now.ldapuser.txt; 
       ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=0-1499' > $now.ldapgroup_0.txt; 
       ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=1500-2999' > $now.ldapgroup_1.txt; 
       ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=3000-4499' > $now.ldapgroup_2.txt; 
       ldapsearch -x -D "user_who_syncs_jira@domain.sample" -w "PASSWORD_HERE" -b "OU=your,OU=group,OU=dn,OU=here" -H ldaps://your-ldap-load-balancer.sample:636 "(YOUR_SAMPLE_GROUP_FILTER_HERE)" 'member;range=4500-6000' > $now.ldapgroup_3.txt;  
       sleep 20m; 
     done;

  2. Execute the following python script against the directory containing the above files, for example: python3 parse.py "/path/to/directory/" 

     parse.py

     import argparse
     import os.path
     from os import listdir
     from os.path import isfile, join
     
     
     def member_line(line: str):
         return line.startswith('member;range')
     
     
     parser = argparse.ArgumentParser(description='Find LDAP duplicates')
     parser.add_argument('path', metavar='path', type=str,
                         help='path to files')
     args = parser.parse_args()
     
     filegroups = {}
     onlyfiles = sorted([f for f in listdir(args.path) if isfile(join(args.path, f))])
     for filename in onlyfiles:
         split_filename = filename.split('.')
         if 'ldapgroup' in split_filename[1]:
             groupname = split_filename[0]
             if groupname not in filegroups:
                 filegroups[groupname] = []
             filegroups[groupname].append(filename)
     for groupname, filenames in filegroups.items():
         print(f'Processing {groupname}')
         occurrences_group = {}
         total_group = 0
         for filename in filenames:
             with open(args.path + filename) as f:
                 lines = f.read().splitlines()
                 occurrences_file = {}
                 for i, line in enumerate(lines):
                     if member_line(line):
                         dn = line.split(":")[1]
                         if i + 1 < len(lines) and not member_line(lines[i+1]):
                             dn += lines[i+1].strip()
                         total_group += 1
                         if dn not in occurrences_file:
                             occurrences_file[dn] = 0
                         occurrences_file[dn] += 1
                 for key, value in occurrences_file.items():
                     if key not in occurrences_group:
                         occurrences_group[key] = 0
                     occurrences_group[key] += value
         duplicates_group = {k: v for k, v in occurrences_group.items() if v > 1}
         if len(duplicates_group) > 0:
             # for duplicate, count in duplicates_group.items():
             #     print(f'Duplicated user {duplicate}, occurrences {count}')
             print(f'Found duplicates in group {groupname}, sample user: {list(duplicates_group.keys())[0]}')
         print(f'Total: {total_group}, Unique member entries: {len(occurrences_group)}')
      

  3. Check the output to the script. In each run, the "Total" should match exactly the "unique member entries".

     1. Example healthy output:(meaning this KB does not apply)

        Processing 1632103058
        Total: 6233, Unique member entries: 6233
        Processing 1632103958
        Total: 6233, Unique member entries: 6233
        Processing 1632104859
        Total: 6234, Unique member entries: 6234
        Processing 1632105760
        Total: 6234, Unique member entries: 6234 ...

     2. Example bad output: (meaning this KB applies)

        Processing 1632085541
        Found duplicates in group 1632085541, sample user:  CN=Test\, user,OU=example,DC=example,DC=com
        Total: 6234, Unique member entries: 6110
        Processing 1632086742
        Found duplicates in group 1632086742, sample user:  CN=User\, Name,OU=example,DC=example,DC=com
        Total: 6234, Unique member entries: 6116
        Processing 1632087943
        Found duplicates in group 1632087943, sample user:  CN=Test\, user,OU=example,DC=example,DC=com
        Total: 6234, Unique member entries: 6221 ...

原因

During the directory syncronization process, a query to fetch the membership of each group is issued to the LDAP server defined in the directory configuration.

If the membership count on the LDAP side of any group is larger than a certain value (in AD, default 1500), only those memberships up to that count are returned by the server, and, Jira must issue a second LDAP search command with an adjusted range to the server. This process repeats until there are no more ranges to fetch.

This process works well in normal situations, and, is an expected part of any LDAP client speaking to a LDAP server. However, if a load balancer in between Jira interferes with this, for example, routing the first query to one server, then the second to a different server, if these two servers have slightly different contents, the range could be offset in a way that duplicates or all together removes results.

This does not affect a load-balancer-less normal ActiveDirectory DNS round robin scenario, where the server name is the simply the domain itself, eg, corp.example.com, as the operating system will resolve the name to an IP, cache it, then return subsequent requests to the same server. The problem is introduced in some load balancers than do not forward subequent queries to the same server.

ソリューション

• Modify the directory within Jira so that it only connects to a single server, eg, a single domain controller, or, the FQDN of the domain, then run a directory full syncronistion within Jira or
• Modify the load balancer configuration to forward subsequent ranged request to the same server

Other notes

• Microsoft Documentation on range retrieval, including limits in Windows Server Active Directory: https://docs.microsoft.com/en-us/previous-versions/windows/desktop/ldap/searching-using-range-retrieval