Git operations on Bitbucket server are very slow due to Scriptrunner plugin
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Git operations such as clone, push, and pull are slow on the Bitbucket server. The operations are very slow even on a smaller repository which is of size few MBs.
Environment
Bitbucket Datacenter 7.17.10
The issue is seen with multiple Git server versions such as 2.27.0, 2.34.1
Git client version 1.8.3.1
The issue is seen in multiple repositories
Diagnosis
Enable Git trace logging on the client side and reproduce the issue by clone/push/pull operation from the client machine:
For Linux
1
2
3
export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1For Windows
1
2
3
set GIT_TRACE_PACKET=1
set GIT_TRACE=1
set GIT_CURL_VERBOSE=1Git Client Logs
In the Git client logs, you would see the push --mirror operation seems stuck at sideband for more than 45 minutes after which the operation was killed:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
Compressing objects: 100% (14351/14351)
Compressing objects: 100% (14351/14351), done.
debug2: channel 0: rcvd adjust 1052672
Writing objects: 97% (45564/46973), 3.07 MiB | 5.37 MiB/s
Writing objects: 98% (46034/46973), 3.07 MiB | 5.37 MiB/s
Writing objects: 99% (46504/46973), 3.07 MiB | 5.37 MiB/s
debug2: channel 0: rcvd adjust 1052672
Writing objects: 100% (46973/46973), 3.07 MiB | 5.37 MiB/s
debug2: channel 0: read<=0 rfd 5 len 0
15:32:36.738462 pkt-line.c:80 packet: sideband< \2Resolving deltas: 100% (19737/19737) \15
remote: Resolving deltas: 100% (19737/19737) [K
15:32:36.738500 pkt-line.c:80 packet: sideband< \2Resolving deltas: 100% (19737/19737), done.
remote: Resolving deltas: 100% (19737/19737), done.[K
15:32:41.972138 pkt-line.c:80 packet: sideband< \1
15:32:46.976448 pkt-line.c:80 packet: sideband< \1
16:17:49.668651 pkt-line.c:80 packet: sideband< \1
16:17:54.671288 pkt-line.c:80 packet: sideband< \1
16:17:59.677102 pkt-line.c:80 packet: sideband< \1
16:18:04.681603 pkt-line.c:80 packet: sideband< \1
16:18:09.686374 pkt-line.c:80 packet: sideband< \1
^Cdebug3: send packet: type 1
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o0/0 e[write]/0 fd -1/8/9 sock -1 cc -1)
Killed by signal 2.Server Profiler logs
From the profiler logs, you will see:
1
2
3
4
5
6
2023-07-05 16:18:09,958 | ssh-scm-request-handler | <Request-ID> | <username> | 1l95sgp
[2734902.1ms] - SSH - git-receive-pack '/<repository-slug>/chat-speed-test02.git'
[4.1ms] - InternalRepository com.atlassian.stash.internal.repository.RepositoryDao.getBySlug(String,String,String)
[11.9ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
[1.5ms] - boolean com.atlassian.bitbucket.scm.ScmRequestCheckService.checkActionAllowed(ScmRequest)
[2734878.8ms] - nio: /usr/local/bin/git receive-pack /var/atlassian/application-data/bitbucket/shared/data/repositories/86651
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
2023-07-05 16:18:09,968 | hook-callback:thread-2 | - | - | -
[2732997.8ms] - DefaultHookService hook callback git-pre-receive for repository 8665
[2732995.7ms] - RepositoryHookResult com.atlassian.bitbucket.hook.repository.RepositoryHookService.preUpdate(RepositoryHookRequest)
[4.5ms] - List com.atlassian.stash.internal.repository.RepositoryHookDao.findByKeys(Collection,Collection,boolean)
[10.6ms] - com.atlassian.bitbucket.server.bitbucket-git-lfs:verify-lfs-locks-hook#preUpdate
[12.3ms] - com.atlassian.bitbucket.server.bitbucket-jira-commit-checker:jira-commit-checker#preUpdate
[4.8ms] - com.atlassian.bitbucket.server.bitbucket-ref-restriction:restrictionEnforcerHook#preUpdate
[11.4ms] - com.isroot.stash.plugin.yacc:yaccGlobalHook#preUpdate
[4.1ms] - RepositoryHook com.atlassian.bitbucket.hook.repository.RepositoryHookService.getByKey(Scope,String)
[3.9ms] [count: 2, avg: 1.9ms] - InternalRepositoryHook com.atlassian.stash.internal.repository.RepositoryHookDao.findByKey(Scope,String)
[4.1ms] [count: 2, avg: 2.0ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
[19.4ms] - com.onresolve.stash.groovy.groovyrunner:mirrored-repos-script-receive-hook#preUpdate
[57.4ms] - com.onresolve.stash.groovy.groovyrunner:script-receive-hook#preUpdate
[9.1ms] - com.atlassian.bitbucket.server.bitbucket-repository-hooks:script-repository-hook#preUpdate
[7.0ms] - List com.atlassian.stash.internal.hook.script.InternalHookScriptService.getSummariesByHookRequest(RepositoryHookRequest,HookScriptType)
[6.7ms] [count: 3, avg: 2.2ms] - Stream com.atlassian.stash.internal.hook.script.HookScriptConfigDao.stream(HookScriptConfigCriteria)
[12.9ms] - com.atlassian.bitbucket.server.bitbucket-repository-hooks:legacy-script-repository-hook#preUpdate
[30.9ms] - nio: /usr/local/bin/git cat-file --batch-check
[3.0ms] - com.atlassian.stash.internal.scm.git.command.refdb.RefsGitRefDbCommand.call()
[149.2ms] - nio: /usr/local/bin/git rev-list --format=%H%x02%P%x02%aN%x02%aE%x02%at%x02%cN%x02%cE%x02%ct%n%B%n%x03END%x04 --ignore-missing --topo-order --no-min-p...
[44.6ms] [count: 2, avg: 22.3ms] - void com.atlassian.bitbucket.commit.CommitService.streamChanges(ChangesRequest,ChangeCallback)
[3.8ms] [count: 2, avg: 1.9ms] - InternalCommitDiscussion com.atlassian.stash.internal.commit.CommitDiscussionDao.findByRepositoryAndCommit(int,String)
[32.8ms] [count: 2, avg: 16.4ms] - nio: /usr/local/bin/git diff-tree -C -r --always --format=%H --root 90779724522b9a407eb62bde3c8fddd33f45a3c7 --
[45.4ms] [count: 2, avg: 22.7ms] - void com.atlassian.bitbucket.commit.CommitService.streamChanges(ChangesRequest,ChangeCallback)
[3.3ms] [count: 2, avg: 1.6ms] - InternalCommitDiscussion com.atlassian.stash.internal.commit.CommitDiscussionDao.findByRepositoryAndCommit(int,String)
[37.5ms] [count: 2, avg: 18.8ms] - nio: /usr/local/bin/git diff-tree -C -r --always --format=%H --root f73f9c4653b68fd4e335ada75ef999924303ba7f --
[37.6ms] [count: 2, avg: 18.8ms] - void com.atlassian.bitbucket.commit.CommitService.streamChanges(ChangesRequest,ChangeCallback)
[3.7ms] [count: 2, avg: 1.8ms] - InternalCommitDiscussion com.atlassian.stash.internal.commit.CommitDiscussionDao.findByRepositoryAndCommit(int,String)
[29.9ms] [count: 2, avg: 14.9ms] - nio: /usr/local/bin/git diff-tree -C -r --always --format=%H --root dc327dde0a6595e0c06cb07eb2d0b06f4b398cb1 --
[44.8ms] [count: 2, avg: 22.4ms] - void com.atlassian.bitbucket.commit.CommitService.streamChanges(ChangesRequest,ChangeCallback)
[3.5ms] [count: 2, avg: 1.8ms] - InternalCommitDiscussion com.atlassian.stash.internal.commit.CommitDiscussionDao.findByRepositoryAndCommit(int,String)
[36.8ms] [count: 2, avg: 18.4ms] - nio: /usr/local/bin/git diff-tree -C -r --always --format=%H --root 095778654ad0a77e0a283f27c6b1768dc59160c3 --
[45.1ms] [count: 2, avg: 22.5ms] - void com.atlassian.bitbucket.commit.CommitService.streamChanges(ChangesRequest,ChangeCallback)
[2.9ms] [count: 2, avg: 1.5ms] - InternalCommitDiscussion com.atlassian.stash.internal.commit.CommitDiscussionDao.findByRepositoryAndCommit(int,String)
[36.4ms] [count: 2, avg: 18.2ms] - nio: /usr/local/bin/git diff-tree -C -r --always --format=%H --root 04794f8fd784c6cebecd93756657fd0ad33b3277 --
[41.7ms] [count: 2, avg: 20.8ms] - void com.atlassian.bitbucket.commit.CommitService.streamChanges(ChangesRequest,ChangeCallback)You would see a lot of "git diff-tree" commands processed against the commits in the span of 45 minutes. These git diff-tree commands are run by either YACC hook or ScriptRunner for Bitbucket plugin as you would see the hooks being called in the push operation.
Cause
In this case, the cause of the issue is the ScriptRunner for Bitbucket plugin.
Solution
Enable Safe mode following the steps on this page, on the Bitbucket server instance (that disables all the 3rd party applications during the safe mode) and see if the push operation takes a long time.
Perform a git push --mirror on the repository and verify if the operation takes long.
If the operation processes immediately, the issue could be due to a third-party plugin.
To identify what plugin is causing the problem, enable one plugin at a time, reproduce the issue, and continue the steps until the plugin is identified.
Disable the plugin and proceed with the push operation
Work with plugin vendor support for further assistance.
Disabling the Scriptunner plugin should help resolve this issue.
Was this helpful?