Bitbucket Server stops responding due to a Java-level deadlock
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Symptoms
Bitbucket Server becomes unresponsive.
The sequence of events that lead to this situation are not yet known and it is not easily reproducible.
Environment
The issue affects instances using Java 8.
Thread dump
Thread dumps taken during this time would show Java-level deadlocks.
The deadlock is encountered between a ClientPoller, BlockPoller and a few other http/s threads on the following methods:
java.nio.channels.spi.AbstractSelectableChannel.removeKey
java.nio.channels.spi.AbstractSelector.cancel
Sample thread dump
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
Found one Java-level deadlock:
=============================
"https-jsse-nio-8201-exec-222":
waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
which is held by "https-jsse-nio-8201-ClientPoller"
"https-jsse-nio-8201-ClientPoller":
waiting to lock monitor 0x00007f8bb41791b8 (object 0x00000006c4342f48, a java.lang.Object),
which is held by "https-jsse-nio-8201-exec-52"
"https-jsse-nio-8201-exec-52":
waiting to lock monitor 0x00007f8c38306148 (object 0x00000006c5f6dcb0, a java.util.HashSet),
which is held by "https-jsse-nio-8201-BlockPoller"
"https-jsse-nio-8201-BlockPoller":
waiting to lock monitor 0x00007f8b441b6568 (object 0x00000006c4343de0, a java.lang.Object),
which is held by "https-jsse-nio-8201-exec-15"
"https-jsse-nio-8201-exec-15":
waiting to lock monitor 0x00007f8c18043f38 (object 0x00000006c6278be8, a java.util.HashSet),
which is held by "https-jsse-nio-8201-ClientPoller"
Java stack information for the threads listed above:
===================================================
"https-jsse-nio-8201-exec-222":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c43842d0> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c43842c0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c43842f8> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c43843a8> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4384308> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-ClientPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c6278be8> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c31a3fc0> (a sun.nio.ch.Util$3)
- locked <0x00000006c31a3fb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c31a3f68> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:706)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-exec-52":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c5f6dcb0> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4342f58> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4342f48> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4342f80> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4343030> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4342f90> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
"https-jsse-nio-8201-BlockPoller":
at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
- waiting to lock <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
- locked <0x00000006c5f6dcb0> (a java.util.HashSet)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c3da4290> (a sun.nio.ch.Util$3)
- locked <0x00000006c3da4280> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c3da4238> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"https-jsse-nio-8201-exec-15":
at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
- waiting to lock <0x00000006c6278be8> (a java.util.HashSet)
at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
- locked <0x00000006c4394e78> (a sun.nio.ch.SelectionKeyImpl)
at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
- locked <0x00000006c4343de0> (a java.lang.Object)
at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
- locked <0x00000006c4394ea0> (a java.lang.Object)
at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:564)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1169)
- locked <0x00000006c4394f50> (a org.apache.tomcat.util.net.SecureNioChannel)
at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:402)
at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:665)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1582)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
- locked <0x00000006c4394eb0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.Cause
The deadlock is caused by the following bug in Tomcat versions 9.0.20 to 9.0.31:
64007 - A Deadlock with Poller run and cancelledKey
Affected Versions
The Bitbucket server versions affected by this bug are:
Bitbucket Version | Embedded Tomcat Version |
|---|---|
6.5.x | 9.0.21 |
6.6.x | 9.0.21 |
6.7.x | 9.0.21 |
6.8.x | 9.0.21 |
6.9.x | 9.0.21 |
6.10.0 - 6.10.2 | 9.0.27 |
7.0.0 - 7.0.1 | 9.0.30 |
Workarounds
Perform a restart of Bitbucket Server.
For multi-node Bitbucket Data Center instances, only the affected node needs to be restarted.
Fix Versions
This issue is fixed in Bitbucket Server versions 6.10.3+, 7.0.2+, 7.1.0+ or later.
These versions use Tomcat version 9.0.31+ or later.
Java 11
If an upgrade to a fix version of Bitbucket is not possible, an alternative is to use Java 11 where the issue with deadlocks is addressed (reference: Tomcat repository commit). Java 11 is installed by default with the Bitbucket 8.6.0 installer.
Updated on April 1, 2025
Was this helpful?
Still need help?
The Atlassian Community is here for you.