JIRA unresponsive due to DB connections getting stuck after DB failover
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After DB failover from DB1 to DB2 instance, the Jira node becomes unresponsive and requires a restart to recover.
Environment
Jira 8.20.x
Postgres DB
Driver version 42.5.1 or prior
Diagnosis
Note that officially DB failover is not supported and there exists a feature request for Support database failover.
There were some fatal DB shutdown error in the logs, however no error pertaining to DB connection were seen later after few hours.
Inspect thread dumps to check the state of the HTTP and Daemon threads if they are BLOCKED. With the
socketTimeoutexplicitly set, the connection still did not timeout.
dbconfig.xml
1
2
3
4
5
<jira-database-config>
<jdbc-datasource>
<connection-properties>tcpKeepAlive=true;socketTimeout=240</connection-properties>
</jdbc-datasource>
</jira-database-config>Cause
Due to potential bug Connection hang during initialisation / enableSSL in the Postgres Database driver, the connection could not be re-established/refreshed and appeared to be in HUNG state, causing other HTTPs and Daemon threads to be blocked and making JIRA unresponsive.
StackTrace from thread dump It indicates that the thread was hung while obtaining DB connection for very long time until JIRA was restarted.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
"localq-reader-15" #92 daemon prio=5 os_prio=0 cpu=1719.21ms elapsed=152037.72s tid=0x00007f8a88ae4000 nid=0x835 runnable [0x00007f89f21ef000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.13/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.13/Unknown Source)
at java.net.SocketInputStream.read(java.base@11.0.13/Unknown Source)
at java.net.SocketInputStream.read(java.base@11.0.13/Unknown Source)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
at org.postgresql.core.PGStream.receiveChar(PGStream.java:443)
at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:505)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:149)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
at org.postgresql.Driver.makeConnection(Driver.java:465)
at org.postgresql.Driver.connect(Driver.java:264)
at java.sql.DriverManager.getConnection(java.sql@11.0.13/Unknown Source)
at java.sql.DriverManager.getConnection(java.sql@11.0.13/Unknown Source)
at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:87)
at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:173)Solution
Upgrade the JDBC driver for PostgreSQL to42.5.2 or later following the steps mentioned below
To do that, download the driver from https://jdbc.postgresql.org/
Stop Jira
Copy the new driver to
<jira-install>/libRemove the old one
Start Jira
Was this helpful?