Cache Replication Jira Data Center Troubleshooting

Summary

Jira keeps some data in memory local to the node, especially general configuration data that are used often, such as permissions. The cache synchronization is asynchronous (7.9 and later) but expected to be fast and consistent across nodes. It is communicated and being replicated over the network.

Solution

Where Cache-related information is stored

In-memory

Jira caches are by default stored in memory, local to each node. This allows the super quick retrieval of data that are seen by Jira to be the most requested.

Local files (7.9 and later)

Data that are stored in the local files in <local-home-directory>/localq are only the cache replication queues and not the actual cache data. Jira Data Center Cache Replication document explains more to how localQ is being used in Jira Data Center.

Information stored in this cache

This cache stores general Jira information, including permissions, users, groups, memberships, and administrative configurations (ie. screen configurations, field configurations, and much more).

Common Problems

As most administrative functions are cached, most problems are detected when an administrative change is performed on one node, but not seen on other nodes. The following are some problems with Cache Replications in Jira Data Center:

• Health Check Cluster Cache Replication indicates a warning or an error.

• Inconsistency of Filter permission between nodes in the same cluster

  • Example: filter A is private in Node-1 while shared to group jira-users in Node-2.

• Inconsistency of Project permission between nodes in the same cluster

• Inconsistency of Field configurations between nodes in the same cluster

  • Example: newly created custom field exists on edit screen in one node, but not in another.

⚠️ Inconsistent field values on issue tickets are not caused by cache replication problems. These are more likely due to indexing problems.

Solutions

In general, if there is a problem with cache replication, it should be detected and resolved by Jira. Sometimes, however, this fails, and there may not be an error. In this case, symptoms like those listed in 'common problems' will be indicators that the cache needs to be rebuilt. There is not direct method to rebuild or clean up the cache in Jira, however the following workarounds help resolve many cache-related problems.

• Reboot the problem node

  • If only one node has an inconsistency, or if one node has out-of-date configuration information, restarting that node may fix the problem. This will not resolve a recurring replication problem.

• Fully shut down and restart the whole cluster.

  • In some cases, rebuilding one node's cache is not enough; other nodes may have 'poison pills' (bad data) they are repeatedly giving to the problem node. This will require a full shutdown to avoid, as this way, when the nodes come online, there is no poisoned cache to retrieve and they must rebuild from scratch.

⚠️ If these solutions resolve the problem temporarily but the problem returns, open a Support Issue with the information detailed below for a more in-depth analysis.

Sending Information to Support

1. Generate Jira support zip of all nodes in the cluster.

   1. For Jira 7.8.x and earlier, additional logging is required. See below.

2. Share the screenshot result of the Jira Data Center Health Check.

3. The output of CLUSTERNODE table.

4. The output of nc -vnz -w 1 <IP> <port> between nodes.

Logging for Cache Replication

For Jira 7.9.x and above. Default logging has been added, see: https://confluence.atlassian.com/enterprise/monitoring-the-cache-replication-954262834.html

For Jira 7.8.x and earlier. Additional logging must be enabled. To get detailed logging of what Jira is doing add a new logger to Jira’s Logging and Profiling page

Logging for Replication sending communication

level: TRACE

package: net.sf.ehcache.distribution

Logging for Replication receiving communication

level: TRACE

package: com.atlassian.jira.cluster.distribution.JiraCacheManagerPeerProvider