JIRA Data Center Asynchronous Cache replication failing health check
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
This article applies to to Jira Data Center documentation and is not relevant for a standard JIRA installation.
Problem
Cluster health check is failing for cache replication and the following stack trace is present in Atlassian-jira.log
1
2
3
4
5
6
7
8
9
{cacheName='', action=REMOVE_ALL, key=null, value=null, creationTimeInMillis=1534168756786}from cache replication queue: [queueId=queue_node1_6_4701d15959342d6659cddeeebe1bdbe7, queuePath=/var/atlassian/jira-home/localq/queue_nodexxxxxx7], failuresCount: 10/10. Removing from queue. Error: java.rmi.NoSuchObjectException: no such object in table
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.NoSuchObjectException: no such object in table
at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:88)
at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:83)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)1
2
localq-reader-3 ERROR [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Abandoning sending: LocalQCacheOp...failuresCount: 1/1. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: node1; nested exception is:
java.net.UnknownHostException: node1Affected Versions:
Jira Data Center versions 7.9 and above with Clustering enabled
Cause
Individual hostnames are not resolvable by other Jira Data Center nodes in the cluster.
This can also occur if there's an invalid hostname set by a JVM parameter:
1
-Djava.rmi.server.hostname=localhost⚠️"-Djava.rmi.server.hostname=" parameter defined inside the "<install-path>/bin/setenv.sh" will override the parameter "ehcache.listener.hostName=" in the file "<jira-home>/cluster.properties"
Resolution
There are 3 possible solutions for this issue:
Remove the offending input from the file "
<install-path>/bin/setenv.sh"Set this parameter value to the machine's IP address;
Set this parameter value to a resolvable hostname;
⚠️JIRA must be restarted for this change to take effect
Was this helpful?