Getting Bad Gateway (502) error in Jira logs during DVCS hourly syncs

Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

While syncing a large GitHub repository Jira admin might notice "Bad Gateway (502)" error recorded in the atlassian-jira.log

The entry point of this issue can be in several places. Most common place where you might discover this issue is: user(s) complaining regarding GitHub Pull Requests (PRs) missing in the Development panel on certain issue key(s)

⚠️ Troubleshooting this error from the entry point is considerably difficult and time-taking process. This KB intends to share the steps Atlassian Support generally takes to identify the problem.

Environment

Jira with DVCS connection to github.com with large repositories. In this specific case, "Large" means a 10s of thousands of closed Pull Requests.

Diagnosis

The atlassian-jira.log will be littered with this Bad Gateway (502) error multiple times during the impacted Github Repository is being sync'd.

2022-11-22 14:30:48,564+0000 http-nio-8080-exec-58 url: /rest/bitbucket/1.0/repository/18073/softsync; user: xxxxxx WARN xxxxxx 867x1548737x6 1yipd76 x.x.x.x,y.y.y.y,z.z.z.z /rest/bitbucket/1.0/repository/18073/softsync [c.a.j.p.d.spi.github.GithubClientWithTimeout] Failed to execute request [GET /repositories/75224144/pulls?state=closed&per_page=100&page=3] errors: 
    Bad Gateway (502)
java.io.IOException: Bad Gateway (502)
	at org.eclipse.egit.github.core.client.GitHubClient.createException(GitHubClient.java:564)
	at org.eclipse.egit.github.core.client.GitHubClient.get(GitHubClient.java:740)
	at com.atlassian.jira.plugins.dvcs.spi.github.GithubClientWithTimeout.lambda$get$0(GithubClientWithTimeout.java:67)
	at com.atlassian.jira.plugins.dvcs.spi.github.GithubClientWithTimeout.executeRequest(GithubClientWithTimeout.java:103)
	at com.atlassian.jira.plugins.dvcs.spi.github.GithubClientWithTimeout.get(GithubClientWithTimeout.java:67)
	at org.eclipse.egit.github.core.client.PageIterator.next(PageIterator.java:173)
	at org.eclipse.egit.github.core.service.GitHubService.getAll(GitHubService.java:151)
	at org.eclipse.egit.github.core.service.PullRequestService.getPullRequests(PullRequestService.java:152)
	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.IssueCommentPayloadEventProcessor.getPullRequestByHtmlUrl(IssueCommentPayloadEventProcessor.java:80)
	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.IssueCommentPayloadEventProcessor.process(IssueCommentPayloadEventProcessor.java:52)
	at com.atlassian.jira.plugins.dvcs.spi.github.service.impl.GitHubEventProcessorAggregatorImpl.process(GitHubEventProcessorAggregatorImpl.java:64)
	at com.atlassian.jira.plugins.dvcs.service.GitHubEventServiceImpl$SyncTransactionCallback.doInTransaction(GitHubEventServiceImpl.java:223)
	at com.atlassian.jira.plugins.dvcs.service.GitHubEventServiceImpl$SyncTransactionCallback.doInTransaction(GitHubEventServiceImpl.java:177)
	at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
	at com.atlassian.jira.DefaultHostContextAccessor.doInTransaction(DefaultHostContextAccessor.java:50)
	..[TRIMMED]..
	at java.lang.Thread.run(Thread.java:750)

ℹ️ As this is a 5xx class of errors, it's clear that it's the error response from the server side i.e., GitHub in this case. In other words, Jira is at the receiving end of the error and not the initiator.

As mentioned earlier, it might take some time to identify and validate what is causing the 502. However, few key pointers to remember to get to the bottom of this are:

Step #1: Are you looking at the correct Repository❓

Jira (DVCS Connector Plugin) registers the GitHub Organizations/Accounts and Repositories in its database with its own IDs. This ID is internal to Jira only and you might not be sure which ID corresponds to which repository. This is why it's important to follow the log trail for the impacted repository. In order to do that:

- First look for the Repository ID in the error message to ensure you're troubleshooting the correct impacted repo. If the entry point was some user complaining about a open/closed PR not showing up in the Development panel, identify which repository the PR belongs to. In the example above, the repository ID is 18073 (/rest/bitbucket/1.0/repository/18073/softsync).
- Since the repo ids are provided by Jira while registering the GitHub Organization/Account into DVCS connector plugin, you should further validate whether this is the correct repo ID under the correct Org where you're having sync issues. You can use the following SQL query to validate by replacing the <REPO_ID> below:

SELECT
    om."NAME" AS "Organization name", rm."NAME" AS "Repository name" 
FROM
    "AO_E8B6CC_REPOSITORY_MAPPING" rm 
	INNER JOIN "AO_E8B6CC_ORGANIZATION_MAPPING" om ON rm."ORGANIZATION_ID" = om."ID"
WHERE rm."ID" = '<REPO_ID>';

💡 Keep the output of both "Organization name" and "Repository name" saved - we'll need those values in the next steps.

Step #2: Which API call is failing with this 502 ❓

Once you're sure the repo ID matches the concerned repo name, next important thing to find out is which HTTP request is receiving this Bad Gateway (502) error. Most likely it will be a GET REST API Call.

- Please note that the API call will not be recorded in every instance of Bad Gateway (502)error in the logs.
- Look for the stack trace immediately followed or preceded by this entry and find the github REST API call which received the error.
- In the example above focus on "Failed to execute request [GET /repositories/75224144/pulls?state=closed&per_page=100&page=3]".

This REST API endpoint is a short version of the actual GitHub endpoint: [GitHub Documentation] : List Pull Requests [GET /repos/{owner}/{repo}/pulls]

- So the actual REST API would be:

GET /repos/<"Organization name">/<"Repository name">/pulls?state=closed&per_page=100&page=3

ℹ️Quick Gotcha: This REST API call is looking for closed Pull requests in paginated way (100 entries per page)

Step #3: Try to Reproduce the issue 💡

During a repository sync, all DVCS plugin is doing is sending a bunch of REST API calls to the DVCS source/provider (in this case GitHub) → Receiving the response data → Storing and Processing the response data in DVCS AO_E8B6CC_* tables. For this particular error, Jira is receiving the 502 error while sending the above REST API call. At this point, we understood that Jira is the victim here and can be eliminated from the potential problem areas.

Let's try to reproduce the issue directly by running the same REST call against the same repository on GitHub. For this you can use tools like cUrl or postman or any other REST client at your disposal. As you'd see in GitHub REST API documentation, to run a REST call you'd need a Bearer token to authenticate the call.

- Instead of creating a new token, let's use the same token that Jira is using to emulate the problem even more closely. You can fetch the API token Jira is using from the database for the repository sync with the <Organization_name> we had found in the output of SQL query in Step #1:

SELECT om."ACCESS_TOKEN" FROM "AO_E8B6CC_ORGANIZATION_MAPPING" om WHERE om."NAME" = '<Organization_name>';

⚠️Note: Very common mistake would be to enter the "Repository name" instead of the "Organization name" in the Query above. Since DVCS Accounts are registered with either the Organization or Account name - we save the Access Token against each Organization and use the same token for all individual repos under the Org/Account.

- Now we should have the GitHub access token Jira is using which looks something like gho_xxxxxxxxx and we'll use the same token to make the same REST API call we identified in Step #2:

❯ curl -D- -X GET 'https://api.github.com/repos/<"Organization name">/<"Repository name">/pulls?state=closed&per_page=100&page=3' --header 'Accept: application/vnd.github+json' --header 'Authorization: Bearer gho_xxxxxxxxx'

Step #4: Repeat the REST API in Step #3 multiple times if you don't receive a 502 response in first attempt

You'd learn in the Cause section why repeating the REST API call multiple times is important in order to reproduce the issue.

Cause

This error is sent by GitHub for any REST API or GraphQL API which exceeds the timeout value of 10 seconds. As you'd find the following in GitHub Documentation regarding Timeouts

Timeouts

If GitHub takes more than 10 seconds to process an API request, GitHub will terminate the request and you will receive a timeout response.

This is why this error is rarely seen and happens especially on repositories with very large git history. In this particular case, the repo had >34K closed PRs in the repo. DVCS connector plugin decided the per_page parameter to be 100 because that's the max value as per github documentation. This is why you might find this error to be intermittent and hence to reproduce the issue you might have to repeat the API call multiple times.

Solution

There's no known solution for this issue. Jira is receiving this 502 from GitHub intermittently because GitHub is failing to respond to the REST API within their own timeout value of 10 seconds. If you run into this problem, please reach out to GitHub support and also partner with us in the investigation.

Updated on April 2, 2025

Was this helpful?

It wasn't accurateIt wasn't clearIt wasn't relevant