Improve the DVCS sync performance by migrating webhooks to the new format

Summary

The purpose of this KB article is to provide different ways to migrate all the webhooks to the new polling type, in order to improve the performance of the sync with the DVCS accounts configured in Jira.

Environment

Jira Software Server / Data Canter 8.14.0 and any higher version.

Symptoms

• High CPU usage may be observed on the Database Server;

• One of the top queries showing high usage is the query below. It means that users create a lot of commits / pull requests and DVCS is not able to process them. This number might be high (thousands):

  SELECT COUNT(*) FROM AO_E8B6CC_MESSAGE messageMapping JOIN AO_E8B6CC_MESSAGE_TAG messageTag ON messageMapping.ID = messageTag.MESSAGE_ID JOIN AO_E8B6CC_MESSAGE_QUEUE_ITEM messageQueueItem ON messageMapping.ID = messageQueueItem.MESSAGE_ID WHERE messageTag.TAG = 'synchronization-repository-xxx' AND messageQueueItem.STATE in ( 'PENDING', 'RUNNING', 'SLEEPING' )

• Recent activities as commits/pull requests are not synchronized with Jira.

Solution

Context behind this article

From Jira Software 8.14.0 onwards, a more efficient way to sync repositories from a Version Control System (VCS) such as Github, Bitbucket and Gitlab was introduced in the Jira DVCS module. This new method involves using a new type of webhooks pointing to the Jira application, and used by the VCS repositories.

The difference between the old type of webhooks and the new type is described below.

Before Jira 8.14.0

• the old type of webhook was in use in these Jira versions, referred to as "polling webhook"

• whenever a commit/branch/pull request was updated in a repository, the old type of webhook would be sent from the VCS to Jira

• in return, Jira would send a request to the DVCS to re-sync the repository

• this method was inefficient, since the Jira application would need to send extra requests to the application, causing network contention and possible sync performance issue

• this is what the flow looks like for the "polling webhook"

  

• the below macro shows the structure of the URLs of the old webhook type, for each type of VCS (Bitbucket, Github and Gitlab):

  Webhook URLs - Jira 8.13.x and below

  • URL of the old webhook type (polling type):

    • for Bitbucket Cloud events:

      • PUSH https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/repo/sync

      • PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/sync

    • for Github Cloud and Server events:

      • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/sync

    • for Gitlab Cloud and Server events:

      • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/repository/<repository id>/gitlab/sync

From Jira 8.14.0 onwards

• a new type of webhook was introduced in this Jira version, referred to as "non-polling webhook"

• with new new type of webhook:

  • whenever a commit/branch is updated in a repository, the new type of webhook is sent to Jira, which already contains the information that Jira needs to update the development panel of the linked issues

  • however, for the pull requests, the Jira application will need to send a request to the external application to get details about it

• the benefit of this type of webhook is that:

  • Jira no longer needs to sync with Github for commits and branches

  • The creation of commits and branches usually represent the majority of the operations made in a VCS application, so this change will reduce the number of requests sent by the Jira application to the external application

• this is what the flow looks like for the "non-polling webhook"

  

• the below macro show the structure of the URLs of the new webhook type, for each type of DVCS (Bitbucket, Github and Gitlab):

  Webhook URLs - Jira 8.14.0 and aboveURL of the new webhook type (non-polling type)

  • for Bitbucket Cloud events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/bitbucket

  • for Github Cloud and Server events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/github

  • for Gitlab Cloud and Server events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/gitlab

The main purpose of this change is to improve drastically the performance of the DVCS account sync performed by the Jira application, especially in the situation where Jira is configured with a very high number of DVCS accounts, with with DVCS accounts containing a high number of repositories to sync.

Problem addressed by this article

For any repository that was added in Jira > ⚙ > Applications > DVCS Accounts before the upgrade to Jira 8.14+, such repository will keep using the old type of webhook even after the Jira upgrade. In other word, the Jira upgrade to 8.14+ will not automatically migrate the webhooks to the new type.

Because of that, it is necessary to re-create all these webhooks (by removing and re-adding manually each repository on the Jira side), in order to ensure that the non-polling webhook type is used. In case there is a very high number of repositories (hundreds or thousands), it will be not realistic to manually disable and re-enable them from the Jira UI.

⚠️ Please note that for any repository that was added to the Jira application after the upgrade to 8.14+, it is not necessary to disable and re-enable it since any newly added repository will be using the new type of webhook.

Solution

Disabling and re-enabling each repository will automatically remove the webhooks on the Github/Gitlab/Bitbucket side, and then create new webhooks, using to the new "non polling" type.

If your Jira application is configured with a small number of repositories, then you might consider disabling and re-enabling each repository 1 by 1 directly from the Jira UI via the page. By disabling and re-enabling each repository, the old type of webhook will be deleted and the new type of webhook will be created for that repository.

Because this can be a manual and tedious task, we recommend applying this method for the most used repositories. The SQL query below can help to find those repositories:

• Make sure to update the START_DATE parameter with a different date stamp to see the most used repos in the last few days (for example the last 10 days).

• The SQL query was written and tested on PostgreSQL, and might need to be amended if Jira is using a different type of database

  select "NAME", "ORGANIZATION_ID", "REPOSITORY_URI", "SLUG" from "AO_E8B6CC_REPOSITORY_MAPPING"where "ID" in (select "REPO_ID" from "AO_E8B6CC_SYNC_AUDIT_LOG"where "SYNC_TYPE" like '%WEBHOOKS%' and "START_DATE" > '2021-05-15 00:00:00.000'group by "REPO_ID" order by count("REPO_ID") desc limit 10);

Recommended Solution Steps - Using the Jira UI

Once you have identified which repositories you want to disable/re-enable, you can follow the steps below:

• Take note of the 1st repository you want to disable/re-enable

• Go to the page Jira > ⚙ > Applications > DVCS Accounts

• Click on the accounts the repository belongs to:

  

• Click on the X sign located on the right side of the repository:

  

  • The 2 following actions will happen:

    • The old webhooks from this repository will be deleted on the VCS side (Github, Gitlab...)

    • The status of all the messages linked to this repository will be set to SLEEPING

• Very important step:

  • Before you add this repository back, go check this repository on the VCS side and verify that the old webhooks have been deleted from this repository

  • If they have not been deleted yet, wait until they get deleted

  • The reason why this step is important is because quickly removing and re-adding a repository might lead to the bug JSWSERVER-21377 DVCS Quick enabling and disabling repository on UI lead to race condition and may lead to faulty state of repository.

• After you verified that the old webhooks have been deleted for the repository, you can re-add it in the Jira UI by searching for that repo in the dropdown Add repository and clicking on the Add button:

  

  • The 2 following actions will happen:

    • The new type of webhooks will be created for this repository on the VCS side (Github, Gitlab...)

    • The status of all the messages linked to this repository will be set to PENDING

• Go check this repository on the DVCS side and verify that the webhooks configured for this repository are now using the new URL format

  Webhook URLs - Jira 8.14.0 and aboveURL of the new webhook type (non-polling type)

  • for Bitbucket Cloud events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/bitbucket

  • for Github Cloud and Server events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/github

  • for Gitlab Cloud and Server events:

    • PUSH and PULL https://<Jira-Base-URL>/rest/bitbucket/1.0/webhook/gitlab

• Repeat the same step for the next repositories on your list

Alternative Solution Steps - Using the VCS UI

An alternative solution consists in going directly to the repository configuration in the DVCS application side:

• Go to the setting page repository and look for the webhook configuration

• Check if it is using the old URL as shown in the macro Webhook URLs - Jira 8.13.x and below of this KB article

• If the URL is using the old format, then update the URL directly there, using the new URL format as shown in the macro Webhook URLs - Jira 8.14.0 and above of this KB article

Getting assistance from Atlassian Support

If you run into any issues, please reach out to Atlassian Support to get assistance.

Disable non-polling

It can be disabled by this dark feature flag set to true:

dvcs.connector.polling.synchronization.enabled=true

When this non-polling feature is disabled, the webhooks work but they will trigger soft sync, meaning that the update will not be done immediately but in some time.