DVCS Connector EAP
This feature has been released in Jira Software 8.14. See release notes.
We're happy to announce an EAP release of the DVCS Connector plugin.
Supported Jira versions
We've done all the testing on Jira 8.5 Long Term Support Release, and this is our recommended version. In general, you can test the DVCS Connector with Jira 8.5, and later.
Installing DVCS Connector EAP
Here are the steps that you'll need to perform, to test DVCS improvements in Jira:
- Download one of the ZIP archives linked in the table above, and extract it.
- In Jira, go to Administration > Manage apps > Manage apps.
- Click Upload app, and upload the following .jar files in the following order:
- oauth2-client-plugin-1.0.4-m01-SNAPSHOT.jar (you don't need this if you chose the ZIP archive for Jira 8.5.9 or 8.10+)
- Make sure that the plugins were installed in the right version and are enabled.
- In the drop-down menu, select All apps.
- Search the plugins by using the names below, open each of them, and ensure they have the right version and are enabled (greyed out means the plugin is disabled):
Atlassian OAuth 2 Client - Plugin
(you don't need this if you chose the ZIP archive for Jira 8.5.9 or 8.10+)
Jira DVCS Connector Plugin
Atlassian Jira - Plugins - Development Integration Plugin
- If a GitLab integration was added then first:
- Go to Jira administration > Applications > DVCS accounts
- Select the options beside the GitLab integration(s) and click Delete
- Otherwise, go to Jira administration > Applications > Versions & licenses.
- Find Jira Software 8.x.x
- Select uninstall
- Ensure you unselect “Also remove my Jira Software license and configuration”
- Once this is complete - on the same page:
- find Jira Software and select to Install
- Go to Jira administration > Manage Apps > Manage Apps and select All Apps in dropdown
- Search for Atlassian OAuth 2 Client - Plugin, click it, and select Uninstall
- Search for the other plugins and ensure the versions are not snapshot versions, and they are enabled (greyed out means disabled):
- Jira DVCS Connector Plugin
- Jira DVCS Connector Atlassian Jira - Plugins - Development Integration
Changes and improvements
This release is focused on solving scaling problems for enterprise customers with a large number or organizations and repositories. Here are changes and improvements that we've made:
New in EAP 2
- GitLab integrations work by URL and group. Enter the target group as the Team or Organization name, and it will pull in repositories only for that group and sub-groups under it.
- Separate integrations for GitLab Cloud and GitLab self-managed.
- Various bug fixes: duplication of data, development custom panel syncs, and performance improvements.
As polling implementations can only scale so far, we were limited in what changes we could make in this area. One of the first things we did was to remove full syncs wherever possible as it was taking too much time to link an organization or repository. The volume of requests required to link a large organization meant we were being rate limited by the DVCS account almost immediately. In response, we applied exponential back off which slowed down the whole process even further.
This also affected webhook events, as we would then poll for commit and PR data, and were subject to the same rate limiting restrictions. With the non-polling webhook, we've eliminated these issues. However, the downside of no longer performing a full sync when linking a repository is that historical data is no longer imported by default.
You can still perform a full sync for individual repositories in the user interface. As full sync can put a lot of load on the DVCS account, we want this to be your decision.
GitHub polling operations have been improved in two key areas:
- We now only fetch repositories that a user is the owner of. Previously, we fetched everything (all public, private, etc. repos), but GitHub only allows registering webhooks for repositories where the user is an owner (and not a collaborator).
We implemented fetching of repository deltas for the hourly sync job. Previously we fetched all repositories, then checked every single one for updates. Now, we make the same API call but sort by the last updated field, then fetch only the repositories with updates in the last hour.
All webhooks are now non-polling. We consume as much data as we can from the payloads from Bitbucket, GitHub, and GitLab, and then write immediately to the database. The exception here is the pull request webhooks where we don't always receive commit data in payloads, so we must poll for it. Given that the volume of PR events will be much lower than push events, this is an acceptable trade-off.
On upgrade, we don't modify any existing webhooks mbut any newly linked repositories will automatically use the new webhook endpoints. If you wish to migrate existing repositories, you can do so by either re-linking individual repositories or organizations. Note that this will not result in data loss as by default we no longer do a full sync when linking organizations - removing organizations/repositories only marks them as deleted in the database, but does not purge any data. Only a full sync can do this and full syncs now can only be initiated via customers via the user interface.
If something goes wrong, you can use the feature flag below to disable the non-polling web-hooks and revert to the polling implementation. This does not require that users modify any web-hooks or re-link any repositories - the non-polling hooks will simply delegate to the legacy implementation so no manual configuration is required.
User interface performance
In the past, users would wait minutes for the DVCS admin page to load due to the sheer volume of data being loaded from the backend. The performance was low, as we were continually polling for updates of the same data.
To resolve these issues, we've made a number of improvements:
- All organizations are collapsed by default and won't load any data until expanded. The data will only be reloaded from the server when an organization is expanded. Also, all requests will be paginated.
- All organizations and repository lists are paginated with a maximum of 100 results per page.
- The user interface element used to search for new repos to link from an organization previously loaded all repos into the browser. This filtering logic has been moved to the server side.
- Adding or deleting organizations, registering webhooks, and refreshing repository lists now happens on a background thread, so users can navigate to/from the admin page and still see the progress updates. This also makes the UI much more responsive when adding new organizations.
As part of this EAP, you can also test GitLab as an account in the DVCS Connector with support for both the polling and non-polling webhooks. If you'd like to use it, you'll need to first integrate GitLab with Jira.
For details, see Integrating Jira with GitLab.