Bitbucket Data Center shows the error "DEADLINE_EXCEEDED"
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
While trying to view a certain file on the Bitbucket UI, you get an error 500 "DEADLINE_EXCEEDED".
Environment
Bitbucket Data Center 8.0+
Mesh
Diagnosis
The following stack trace can be seen in the Bitbucket application logs:
1
2
3
4
5
6
7
8
2024-02-01 03:45:40,164 ERROR [http-nio-7990-exec-7] eppds *SUJXMJx225x1112451x0 16swqzb 10.x.xx.xxx,10.x.xx.xxx "GET /projects/EIT/repos/ei-monorepo/browse/agfa-automation-lib/src/test/resources/dbData.xml HTTP/1.1" c.a.s.i.w.HandlerExceptionResolverImpl command timed out
com.atlassian.bitbucket.scm.CommandTimeoutException: command timed out
at com.atlassian.stash.internal.scm.git.mesh.DefaultErrorTranslator.translateIfKnownCause(DefaultErrorTranslator.java:277)
at com.atlassian.stash.internal.scm.git.mesh.DefaultErrorTranslator.maybeTranslate(DefaultErrorTranslator.java:57)
at java.base/java.lang.Thread.run(Thread.java:829)
... 364 frames trimmed
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.998226964s. [closed=[], open=[[remote_addr=localhost/127.0.0.1:7777]]]
at io.grpc.Status.asRuntimeException(Status.java:535)The following stack trace can be seen in the Mesh application logs:
1
2
3
4
5
6
2024-02-01 03:45:30,046 DEBUG [grpc-server:thread-9448] eppds 41FUJOOMx225x19954x2 *SUJXMJx225x1112451x0 127.0.0.1 "ContentService/GetNodeType" (>1 <0) c.a.b.m.r.DefaultRpcRepositoryAccessTracker all has 1 active requests
2024-02-01 03:45:30,154 DEBUG [grpc-server:thread-9448] eppds 41FUJOOMx225x19954x2 *SUJXMJx225x1112451x0 127.0.0.1 "ContentService/GetNodeType" (>1 <0) c.a.b.m.p.nu.StdioNuProcessHandler 6744: Dropping [git ls-tree -z refs/heads/master:agfa-automation-lib/src/test/resources -- dbData.xml] callback invocation (Canceled: true; Timed out: false)
2024-02-01 03:45:30,155 DEBUG [grpc-server:thread-9448] eppds 41FUJOOMx225x19954x2 *SUJXMJx225x1112451x0 127.0.0.1 "ContentService/GetNodeType" (>1 <0) c.a.b.m.p.nu.StdioNuProcessHandler 6744: Dropping [git ls-tree -z refs/heads/master:agfa-automation-lib/src/test/resources -- dbData.xml] callback invocation (Canceled: true; Timed out: false)
2024-02-01 03:45:30,156 DEBUG [grpc-server:thread-9448] eppds 41FUJOOMx225x19954x2 *SUJXMJx225x1112451x0 127.0.0.1 "ContentService/GetNodeType" (>1 <0) c.a.bitbucket.mesh.process.OsProcess 6744: [git ls-tree -z refs/heads/master:agfa-automation-lib/src/test/resources -- dbData.xml] exited 0 in 108ms (stdin: 0, stdout: 64, stderr: 0)
2024-02-01 03:45:47,583 DEBUG [grpc-async:thread-11] eppds 41FUJOOMx225x19955x2 *SUJXMJx225x1112451x0 127.0.0.1 "CommitService/GetCommits" (>1 <0) c.a.bitbucket.mesh.process.OsProcess 6745: [git log --format=commit %H%n%H%x02%P%x02%aN%x02%aE%x02%at%x02%cN%x02%cE%x02%ct%n%B%n%x03END%x04 -2 --no-min-parents --stdin --do-walk --follow --name-status refs/heads/master -- agfa-automation-lib/src/test/resources/dbData.xml] exited 0 in 17419ms (stdin: 18, stdout: 453, stderr: 0)gRPC requests can have deadlines set on them (where after the deadline has passed, the client throws the
DEADLINE_EXCEEDEDerror).We set explicit deadlines on some requests, but the GrpcCommitClient#getCommits call doesn't (
getStubWithDeadlineoverload that doesn't specify atimeout, which falls through tonull).Not specifying an explicit deadline causes the system to look for any timeouts specified on the message, else use the default.
10 seconds is then always added as the default padding, which is the timeout we're hitting here.
Run the failing Git commands on the filesystem
To check how much time the Git commands take when run directly on the Bitbucket instance, run the failing git commands on the Bitbucket filesystem to see how much time it takes.
Go to the Bitbucket filesystem and go to the repository path (the repository ID can be found fromthis doc) and run the following commands:
1
git ls-tree -z refs/heads/master:agfa-automation-lib/src/test/resources -- dbData.xmlSend the commit ID you see in the output of the above command tostdin:
1
echo '<commit-ID>' > stdinNext run the following command, we will send the output ofstdinto the below command:
1
git --no-pager log --format=commit %H%n%H%x02%P%x02%aN%x02%aE%x02%at%x02%cN%x02%cE%x02%ct%n%B%n%x03END%x04 -2 --no-min-parents --stdin --do-walk --follow --name-status refs/heads/master -- agfa-automation-lib/src/test/resources/dbData.xml < stdin > test.txtCause
This issue can happen when the repository size is huge and there are a large number of commits involved and because of the hard-coded timeout of 10 seconds.
Solution
The commit graph is an auxiliary data structure and can be regenerated from the actual commit data with the command "git commit-graph write --reachable". Rewriting commit graphs helps to improve the performance of the git log command.
Follow the below steps for rewriting the commit graphs:
Create a backupof the affected repository in the Bitbucket container and in your local machine as well.
Delete the existing commit graph first, from the affected repositoryon your Bitbucket container. It would be
<Bitbucket-home>/shared/data/repositories/<repository-id>/objects/info/commit-graph. You can find the repository ID by following this documentRun the below commands on the affected repositoryon the Bitbucket host
1
2
cd /var/atlassian/application-data/bitbucket/shared/data/repositories/<repo-ID>
git commit-graph write --reachable --changed-pathsCheck if the issue still persists while viewing the problematic file in the affected repository.
Was this helpful?