After restarting Jira Data Center nodes, the service is running but the node cannot be accessed through the browser
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
After restarting Jira Data Center nodes, the Jira server instance is running on the nodes, but the web interface is not accessible either locally or via the proxy. When attempting to access the URL, the browser will just show a spinning circle but never display anything on the page.
Environment
9.3.0
.
Diagnosis
Within the catalina.out logs when attempting to start the Jira server instance, the start up becomes unresponsive at the point of 'Starting the Servlet engine':
1
2
3
4
5
10-Dec-2022 19:03:44.821 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-80"]
10-Dec-2022 19:03:44.868 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-443"]
10-Dec-2022 19:03:45.321 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [2786] milliseconds
10-Dec-2022 19:03:45.384 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
10-Dec-2022 19:03:45.384 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.65]Collecting a HAR file from the browser when attempting to access the URL will reveal within the 'Timings' tab that the Connection Start is stalled:
Inside of the atlassian-jira.log file it will appear that the instance has started up and is functioning, but you will see messages such as this in the logs:
1
2022-12-10 19:30:00,706-0800 localq-reader-1 WARN [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-INVALIDATION] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='UpmPluginLicenseCache', action=REMOVE_ALL, key=null, value == null ? true, replicatePutsViaCopy=false, creationTimeInMillis=1670728030282} from cache replication queue: [queueId=queue_node2_1_78882aaeb08e9a4c81687b5de2add74f, queuePath=<Jira Home>\localq\queue_node2_1_78882aaeb08e9a4c81687b5de2add74f], failuresCount: 1/2, error: java.rmi.NotBoundException: UpmPluginLicenseCacheCause
The reason that the instance is not accessible via the web browser is that the Tomcat engine has not initialized the web servlet. The Tomcat engine is running (which is why the Jira instance appears to be running) however the component of Tomcat that handles the web requests is not initialized. This can happen due to corruption in the local node cache that is replicated to the other nodes upon start up.
Solution
To resolve this issue:
Stop Jira on all nodes.
Within each nodes <Local Jira Home> directory back up the contents within the localq directory.
Once the contents have been backed up, delete the contents from the localq directory.
⚠️ Do not delete the localq directory itself, just the contents inside.
The contents for this directory will be rebuilt upon starting the Jira service.
Start Jira on each node one by one.
Was this helpful?