要約

Whenever a log message containing “A gap was detected in the pull request history“ appears in the logs, it's likely that you have been affected by a pull-request rescoping bug.

BSERV-11511 - Getting issue details... STATUS

The actual problem will have happened well before this log message and can be detected earlier than that, and this log message is just a consequence of this earlier problem.

What is pull-request rescoping?

Simply put, whenever a change is pushed to a repository, Bitbucket needs to go through all of the pull-requests and perform what's called 'pull-request rescoping'.

This rescoping process includes the following actions taken on each PR:

• Understand whether the pushed change is applicable to that given PR and if so update the PR to include/reflect that new commit. 
• Recalculate whether the PR will be successful against the current state of both the source and the target branch associated with that PR.

So what's the problem?

Sometimes the thread that's running this rescoping process can fail to run or complete successfully for various reasons - which means that until the next rescope operation is attempted, pull requests may not reflect a recent commit/change that was pushed to the repository. Historically - these issues have been inconsistently reproducible, and oftentimes are only investigated well after the rescoping process had already failed, making them very difficult to diagnose and fix.

 However, within some of our latest updates to Bitbucket Data Center, we have fixed numerous subtle bugs and race conditions in pull request rescoping. We are fairly confident that rescoping related problems should be occurring very rarely now. In addition to these fixes, we also improved debug logging to further diagnose these problems when they occur - and have provided detailed diagnosis/troubleshooting steps below in case the issue occurs again for any team.

環境

• Bitbucket Data Center 6.10.7+, 7.6.1+, 7.7.0+

ソリューション

So far we have uncovered two major causes of this bug:

1. Data inconsistency between the database and Git. A pull request tracks the commit IDs of the involved branches in the database. If they become out of sync with what’s actually in the Git repository on disk, it will cause problems. It’s normal for these two values to remain out of sync for a certain amount of time after a change until the pull request rescoping process has brought them back in sync.
   
   If, however, a rescope process failed to bring them in sync, this can cause the following problems:
   
   

   • Misattribution of changes and the log message: “A gap was detected in the pull request history”

   • A pull request becoming un-mergeable, with a message being displayed: “New changes were pushed to <branch> in <project>/<repository> while the merge was being performed. Please retry the merge.”
     
     

2. A plugin or Bitbucket itself publishing an incorrectly calculated RepositoryRefsChangedEvent. This will also cause a message “A gap was detected in the pull request history” in the logs, but otherwise functionality should not be affected.

Debug logs

To make sure debug logs are captured, your team should enable the relevant debug option in the ‘Logging and profiling’ page in the admin section.

Additional trace logging should be enabled when rescoping problems occur frequently, as this can help our team's investigations.

 The following loggers should be set to trace:

• • com.atlassian.stash.internal.pull.rescope.DefaultPullRequestRescopeService

  • com.atlassian.stash.internal.hook.SystemPostRepositoryHook

• If the above DEBUG logs don't provide sufficient information for troubleshooting why the Pull Request Rescoping isn't happening correctly, only then enable the TRACE logging below, as it can be very verbose.
  

  • com.atlassian.stash.internal.concurrent.ClaimTasksFromBucketProcessor

  • com.atlassian.stash.internal.concurrent.HazelcastBucketedExecutor

For more information on how to set these at runtime, or permanently on startup - we recommend reviewing the full details in the following article: Bitbucket Server debug logging.

Early detection

It’s possible to detect problems with pull requests using a script.

When running this script, inconsistent pull request IDs and refs will be printed. It’s possible that pull requests remain inconsistent for a short period of time, however long the pull request rescope process takes.

While pull request data is inconsistent, affected pull requests can not be merged and the message “New changes were pushed to <branch> in <project>/<repository> while the merge was being performed. Please retry the merge.” will be shown when an attempt is made.

import requests
import logging

logging.basicConfig(level=logging.DEBUG)

project = 'PROJECT'
repo = 'repo'
user = 'admin'
password = 'admin'
base_url = 'https://stash.example.com'


def request(req):
    data = requests.get(base_url + '/' + req, auth=(user, password)).json()
    return data


def stream_page(req):
    start=0
    is_last_page=False
    while not is_last_page:
        data = request(req + f'&start={start}')
        is_last_page = data['isLastPage']
        if not is_last_page:
            start = data['nextPageStart']
        for value in data['values']:
            yield value


if __name__ == '__main__':
    branch_map = {}
    for branch in stream_page(f'rest/api/latest/projects/{project}/repos/{repo}/branches?limit=1000'):
        branch_map[branch['displayId']] = branch['latestCommit']

    for pr in stream_page(f'rest/api/latest/projects/{project}/repos/{repo}/pull-requests?limit=1000'):
        id = pr['id']
        if pr['fromRef']['latestCommit'] != branch_map[pr['fromRef']['displayId']]:
            print('PR #' + str(id) + ' fromRef mismatch: ' +
                  pr['fromRef']['latestCommit'] +
                  ' != ' +
                  branch_map[pr['fromRef']['displayId']])
        if pr['toRef']['latestCommit'] != branch_map[pr['toRef']['displayId']]:
            print('PR #' + str(id) + ' toRef mismatch: ' +
                  pr['toRef']['latestCommit'] +
                  ' != ' +
                  branch_map[pr['toRef']['displayId']])

Temporary resolution for “un-mergeable” pull requests

If pull requests become un-mergeable because of the message “New changes were pushed to <branch> in <project>/<repository> while the merge was being performed. Please retry the merge.”, an HTTP request can be made to force rescoping again for a certain repository. This will rescope pull requests and make them mergeable again.

curl -v -u username -X POST -d '{"branches":[]}' -H "Content-Type: application/json" "http://example.com/projects/<PROJECT_KEY>/repos/<REPOSITORY>/jobs/rescope-pull-requests"