要約

この記事の背景情報

From Jira Software 8.14.0 onwards, a more efficient way to sync repositories from a Version Control System (VCS) such as Github, Bitbucket and Gitlab was introduced in the Jira DVCS module. This new method involves using a new type of webhooks pointing to the Jira application, and used by the VCS repositories.

The difference between the old type of webhooks and the new type is described below.

Before Jira 8.14.0

• the old type of webhook was in use in these Jira versions, referred to as "polling webhook"
• whenever a commit/branch/pull request was updated in a repository, the old type of webhook would be sent from the VCS to Jira
• in return, Jira would send a request to the VCS to re-sync the repository
• this method was inefficient, since the Jira application would need to send extra requests to the application, causing network contention and possible sync performance issue
• this is what the flow looks like for the "polling webhook"
  

• the below macro shows the structure of the URLs of the old webhook type, for each type of VCS (Bitbucket, Github and Gitlab):

  Webhook URLs - Jira 8.13.x and below

  • URL of the old webhook type (polling type):

    • for Bitbucket Cloud events:

      • PUSH https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/repo/sync

      • PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/sync

    • for Github Cloud and Server events:

      • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/sync

    • for Gitlab Cloud and Server events:

      • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/gitlab/sync

From Jira 8.14.0 onwards

• a new type of webhook was introduced in this Jira version, referred to as "non-polling webhook"
• with new new type of webhook:
  • whenever a commit/branch is updated in a repository, the new type of webhook is sent to Jira, which already contains the information that Jira needs to update the development panel of the linked issues
  • however, for the pull requests, the Jira application will need to send a request to the external application to get details about it
• the benefit of this type of webhook is that:
  • Jira no longer needs to sync with Github for commits and branches
  • The creation of commits and branches usually represent the majority of the operations made in a VCS application, so this change will reduce the number of requests sent by the Jira application to the external application
• this is what the flow looks like for the "non-polling webhook"
  

• the below macro show the structure of the URLs of the new webhook type, for each type of VCS (Bitbucket, Github and Gitlab):

  Webhook URLs - Jira 8.14.0 and above

  URL of the new webhook type (non-polling type)

  • for Bitbucket Cloud events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/bitbucket

  • for Github Cloud and Server events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/github

  • for Gitlab Cloud and Server events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/gitlab

The main purpose of this change is to improve drastically the performance of the DVCS account sync performed by the Jira application, especially in the situation where Jira is configured with a very high number of DVCS accounts, with with DVCS accounts containing a high number of repositories to sync.

Problem addressed by this article

For any repository that was added in Jira > ⚙ > Applications > DVCS Accounts before the upgrade to Jira 8.14+, such repository will keep using the old type of webhook even after the Jira upgrade. In other word, the Jira upgrade to 8.14+ will not automatically migrate the webhooks to the new type.

Because of that, it is necessary to re-create all these webhooks (by removing and re-adding manually each repository on the Jira side), in order to ensure that the non-polling webhook type is used. In case there is a very high number of repositories (hundreds or thousands), it will be not realistic to manually disable and re-enable them from the Jira UI.

Please note that for any repository that was added to the Jira application after the upgrade to 8.14+, it is not necessary to disable and re-enable it since any newly added repository will be using the new type of webhook.

この記事の目的

The purpose of this KB article is to provide different ways to migrate all the webhooks to the new polling type, in order to improve the performance of the sync with the DVCS accounts configured in Jira.

環境

Jira Software Server / Data Canter 8.14.0 and any higher version.

症状

• High CPU usage may be observed on the Database Server;
• One of the top queries showing high usage is the query below. It means that users create a lot of commits / pull requests and DVCS is not able to process them. This number might be high (thousands):

SELECT COUNT(*) FROM AO_E8B6CC_MESSAGE messageMapping 
JOIN AO_E8B6CC_MESSAGE_TAG messageTag ON messageMapping.ID = messageTag.MESSAGE_ID 
JOIN AO_E8B6CC_MESSAGE_QUEUE_ITEM messageQueueItem ON messageMapping.ID = messageQueueItem.MESSAGE_ID 
WHERE messageTag.TAG = 'synchronization-repository-xxx' AND messageQueueItem.STATE in ( 'PENDING', 'RUNNING', 'SLEEPING' )  

• Recent activities as commits/pull requests are not synchronized with Jira.

ソリューション

Disabling and re-enabling each repository will automatically remove the webhooks on the Github/Gitlab/Bitbucket side, and then create new webhooks, using to the new "non polling" type.

There are different ways to do that:

• manually, via the Jira UI
• automatically, using a python script

Manual Solution

If your Jira application is configured with a small number of repositories, then you might consider disabling and re-enabling each repository 1 by 1 directly from the Jira UI via the page. By disabling and re-enabling each repository, the old type of webhook will be deleted and the new type of webhook will be created for that repository.

Because this can be a manual and tedious task, we recommend applying this method for the most used repositories. The SQL query below can help to find those repositories:

• Make sure to update the START_DATE parameter with a different date stamp to see the most used repos in the last few days (for example the last 10 days).

• The SQL query was written and tested on PostgreSQL, and might need to be amended if Jira is using a different type of database
  

  select "NAME", "ORGANIZATION_ID", "REPOSITORY_URI", "SLUG" from "AO_E8B6CC_REPOSITORY_MAPPING"
  where "ID" in (
  select "REPO_ID" from "AO_E8B6CC_SYNC_AUDIT_LOG" 
  where "SYNC_TYPE" like '%WEBHOOKS%' and "START_DATE" > '2021-05-15 00:00:00.000'
  group by "REPO_ID" order by count("REPO_ID") desc limit 10);

Recommended Solution Steps - Using the Jira UI

Once you have identified which repositories you want to disable/re-enable, you can follow the steps below:

• Take note of the 1st repository you want to disable/re-enable
• Go to the page Jira > ⚙ > Applications > DVCS Accounts
• Click on the accounts the repository belongs to:
  
• Click on the X sign located on the right side of the repository:
  
  
  • The 2 following actions will happen:
    • The old webhooks from this repository will be deleted on the VCS side (Github, Gitlab...)
    • The status of all the messages linked to this repository will be set to SLEEPING
• Very important step:
  • Before you add this repository back, go check this repository on the VCS side and verify that the old webhooks have been deleted from this repository
  • If they have not been deleted yet, wait until they get deleted
  • The reason why this step is important is because quickly removing and re-adding a repository might lead to the bug JSWSERVER-21377 - Getting issue details... STATUS
• After you verified that the old webhooks have been deleted for the repository, you can re-add it in the Jira UI by searching for that repo in the dropdown Add repository and clicking on the Add  button:
  
  
  • The 2 following actions will happen:
    • The new type of webhooks will be created for this repository on the VCS side (Github, Gitlab...)
    • The status of all the messages linked to this repository will be set to PENDING
• Go check this repository on the VCS side and verify that the webhooks configured for this repository are now using the new URL format (as listed in the macro Webhook URLs - Jira 8.14.0 and above of this article)
• Repeat the same step for the next repositor(ies) on your list

Alternative Solution Steps - Using the VCS UI

An alternative solution consists in going directly to the repository configuration in the VCS application side:

• Go to the setting page repository and look for the webhook configuration
• Check if it is using the old URL as shown in the macro Webhook URLs - Jira 8.13.x and below of this KB article
• If the URL is using the old format, then update the URL directly there, using the new URL format as shown in the macro Webhook URLs - Jira 8.14.0 and above of this KB article

Automated Solution using a script

If your Jira application is configured with a high number of repositories and manually disabling and re-enabling from the UI is not an option, you should consider this solution.

This solution consists in using a python script which will:

• Connect to the Jira instance
• Depending on the script:
  • Either iterate through every repository on the Jira instance (configured in ⚙ > Applications > DVCS Accounts), and disable and then re-enable the repository
  • Or iIterate through every repository from one specific organization, and disable and then re-enable the repository

要件

• Jira 8.14 and above.
• A service account with Jira administrator rights that can be logged into with basic auth in the Jira application
• This same user should also be an admin on the VCS side (Github, Gitlab, or Bitbucket)
• screen/tmux
• python3

• python requests library. To install, run the following command:

  python3 -m pip install requests

Solution Steps

Scenario 1 - Script to Refresh all repositories in the Jira instance

• Schedule a Jira maintenance window, preferably during the weekend, due to the fact the script might take hours to complete (see the Notes section below)
• Download the python script refresh-all-repositories.py  from refresh-all-repositories.ps1.zip    
• Create a screen or tmux session

• Execute the command below after replacing

  • <jira_base_url> with your Jira application Base URL

  • <username> with the username of a Jira System Administrator

  • Command:

    python3 refresh-all-repositories.py <jira_base_url> <username>

• You'll be prompted for the password for the user you've declared. Type it in and hit enter to kick off the webhook migration
• Detach from the screen or tmux session and let it run.

Example of output from the script:

python3 refresh-all-repositories.py https://jira.my-company.com admin
Password:
Successfully restarted 2 out of 2 repositories.

Scenario 2 - Script to Refresh all repositories in a specific organization

• Schedule a Jira maintenance window, preferably during the weekend, due to the fact the script might take hours to complete (see the Notes section below)
• Download the python script refresh-all-repositories-in-org.py from refresh-all-repositories.ps1.zip
• Create a screen or tmux session

• Execute the command below after replacing

  • <jira_base_url> with your Jira application Base URL

  • <username> with the username of a Jira System Administrator
  • <organization-ID> with the ID of the organization, which can be found in the database table AO_E8B6CC_ORGANIZATION_MAPPING

  • Command:

    python3 refresh-all-repositories-in-org.py <jira_base_url> <username> <organization-ID>

• You'll be prompted for the password for the user you've declared. Type it in and hit enter to kick off the webhook migration
• Detach from the screen or tmux session and let it run.

Example of output from the script:

python3 refresh-all-repositories-in-org.py https://jira.my-company.com admin 1
Password:
Successfully restarted 2 out of 2 repositories.

Notes about these scripts

• Running these scripts take about 3 hours for every 10k repositories, more if there is significant network latency. For this reason, we highly recommend to run them during a maintenance window
• Please ensure that the box you are running this on won't go to sleep. For very large counts of repos on the instance it would be best to run in a screen or tmux session so you can detach and reattach without interfering with the process.
• Results are dumped into the file refresh-all-repositories.log
• If the script is stopped at any point and then started again, it will start over from the beginning.

Getting assistance from Atlassian Support

If you run into any issues while following either method, please reach out to Atlassian Support to get assistance.

Disable non-polling

It can be disabled by this dark feature flag set to true:

dvcs.connector.polling.synchronization.enabled=true

When this non-polling feature is disabled, the webhooks work but they will trigger soft sync, meaning that the update will not be done immediately but in some time.