How to Use Jira Stats Logs for Server Monitoring and Troubleshooting
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Jira has a number of statistics written to the primary Jira log file (atlassian-jira.log*). Starting with Jira 10.4.0, these statistics are now written to a dedicated log file (atlassian-jira-stats.log*). These are logs which contain aggregated information of what is happening in Jira, which is expensive to log for every event, but may be very useful when determining how crucial Jira modules (like: indexing, index propagation, node to node communication, DB access, IO) are performing.
By default Jira is logging stats every 5 minutes. These logs may have different formats but share the same concepts:
total stats - These are statistics gathered from the time the Jira instance was started up. Usually you want to check the last "total" statistics for a given feature to have a general overview of how things are performing
snapshot - These are the statistics gathered for every snapshot which normally covers a five-minute period (please see below how this time interval can be configured using a new property introduced in Jira 8.16). Usually when you find some values which need more investigation in "total" stats, you will try to find the "snapshot" were the interesting event happened. This will narrow down the search for the problematic event to a five-minute period.
Solution
Types of stats logs
Note that Jira stats are being added incrementally since Jira 8.0. The list below describes the latest set of available statistics.
It is also important to note that starting with Jira 8.16 all statistic logs have a common prefix, which can be identified as [JIRA-STATS].
Stats prefix | Description | Since | More |
[LOCALQ] [VIA-INVALIDATION] | Stats related to replicating messages from caches replicated via invalidation | LOCALQ since Jira 7.9 VIA-INVALIDATION since Jira 8.12 topN caches metric since Jira 8.13.4 | |
[LOCALQ] [VIA-COPY] | Stats related to replicating messages from caches replicated via copy | LOCALQ since Jira 87.9 VIA-COPY since Jira 8.12 topN caches metric since Jira 8.13.4 | |
[DBR] [SENDER] | Stats from the process generating DBR messages. This is just after we produced the document and before we put it into Lucene. And before it sends the message to LocalQ. | Jira 8.12 | |
[DBR] [RECEIVER] | Stats from the process receiving DBR messages from other nodes though the network. It will store those messages to Lucene right away. | Jira 8.12 | |
[VERSIONING] | Stats related to issue, comments and worklog versioning.
Measures the impact of the extra DB calls for do the maintenance of the versions for the
issue, comments and worklog. | Jira 8.10 | |
[index-writer-stats] | Write operations for the indexer (aka Lucene). Placing the created Docs into the Lucene (disk). Jira produces the documents in the different places and places them into the Queue, then single thread process Queue and the writing docs to the index (this will be conditional update). | Jira 8.10 | Troubleshooting disk I/O bottlenecks In Jira Data Center Background reindex is slow after upgrading to Jira 8.10 and later |
[indexing-stats] | Stats focusing on the cost of producing the CustomField values (aka FieldIndexers). | Jira 8.1 | |
[lucene-stats] | Measure the Lucene index flush (not related to commit), including Lucene buffer. During foreground indexing index flushes are only caused by Lucene buffer overflow. Opening new searchers requires the flush operation. | Jira 8.0 | |
[INDEX-REPLAY] [STATS] | Node Index Replay is the process responsible for replaying index operation from other nodes. | Jira 8.10 | |
[replication-stats] | Stats measuring the latency of replaying index operations from other nodes. | Jira 8.08.8 | |
[plugin-transaction] | Stats related to how plugin lifecycle events are triggering cache resets. | Jira 8.2 | JRASERVER-64908 - UPM actions may flush internal caches, leading to performance problems |
[OFBIZ-PROPERTY-ENTRY-STORE-STATS] | Stats related to accessing the common property store cache. | Jira 8.14 | JRASERVER-70519 - Performance degradation due to contention in CachingOfBizPropertyEntryStore cache |
[INDEX-FIXER] | Stats related to the server component responsible for making sure the index is consistent. | Jira 8.12 | |
[INDEXING-QUEUE] | Stats about the state to the indexing queue. This is the part of Jira between the threads building issue (and comments, worklogs, ...) Lucene documents and adding/updating the local index with those documents. | Jira 8.16 / Jira 8.13.6 | |
[JIRA-RMI-AUTH] | Stats related to the Jira Data Center cluster authentication for node-to-node cache replication RMI connection creation. | Jira 8.17 / Jira 8.13.8 / Jira 8.5.16 | |
[index-searcher-stats] | Stats related to Lucene's index searchers (searching and creating searchers). | Jira 8.19 | |
[UserSearchServiceStats] | Stats related to find top mentionable & assignable searcher. | Jira 8.19.1 | |
[LOGIN-STATS] [LOGIN-STORE-STATS] [GROUP-DAO-STATS] | Stats related to user log-in. | Jira 8.22.0 | |
[INDEXING-LIMITS] | Stats related to safeguards in indexing issue-related entities. | Jira 8.22.2 / Jira 9.0.0 | |
[NODE-START] | Stats related i ndex management on startup. | Jira 9.1.0 | |
[AvatarManagerStats] | Avatar stats. | Jira 9.6.0 | |
[CacheStats] | Jira-stats for clear cache operations | Jira 9.6.0 | |
[DVCS-MESSAGE] | Stats related to internal event processing system. Every event is presented as message, placed in queue and consumed. | Jira 9.8.0 | |
[DVCS-HTTP] | Stats related to http traffic which is deducted to two separate streams - inbound and outbound traffic. | Jira 9.8.0 | |
[DVCS-PERFORMANCE] | Stats related to performance of synchronisation with git data done by DVCS plugin. | Jira 9.8.0 | |
[AUTOMATION-EVENT] | Stats related to the automation queue, including events added, claimed for processing, and processed. | Jira 10.2.0 | |
[AUTOMATION-RULE] | Stats related to the number of rules executed in the automation queue. | Jira 10.2.0 | |
[StreamAttachmentStoreStats] | Stats related to attachment streaming, including TTFB, transfer rates, content sizes, and file store configuration. | Jira 9.11.0 | |
[IndexSnapshotOperatorStats] | Stats related to snapshot creation in indexing, including attempts, successes, errors, blocked retries, polling, and waiting times. | Jira 9.7.0 | |
[IndexUtilsStats] | Stats related to index snapshot creation, including creation time, compression time, file sizes, copy and deletion times, and parallel creation settings. | Jira 9.7.0 | |
[SecureDefaultsStats] | Stats related to secure defaults, including allowed and disallowed request processing times for filters, servlets, actions, and plugin servlets. | Jira 10.0.0 | |
[LabelSuggesterStats] | Stats related to label suggestions, including settings, collection times, searched terms, and counts for popular labels, as well as timeout metrics for label collection processes. | Jira 10.0.0 |
Troubleshooting performance with Jira Stat
Please review the related KB article to check how stats-logs (metrics) can be used to troubleshoot different performance problems: Troubleshooting performance with Jira Stats
Statistics Interval Configuration
The below new property was introduced in Jira 8.16 and allows to change the time the statistics are taken and written to the Jira log files.
We strongly recommend to leave the stats frequency as per its default shown below. Since Jira 8.16 the period can be configured via a system property and changed to: 15, 30 or 60 minutes.
-Dcom.atlassian.jira.stats.logging.interval=interval5min (default)
-Dcom.atlassian.jira.stats.logging.interval=interval15min
-Dcom.atlassian.jira.stats.logging.interval=interval30min
-Dcom.atlassian.jira.stats.logging.interval=interval60minNote that this system property has to be set on every node and it requires a rolling restart of all nodes.
The log files with a star "*" in the end indicate that the logs roll over. For example, the latest file is atlassian-jira.log, then atlassian-jira.log.1, then atlassian-jira.log.2 and so on.Most logging is managed by Log4j via the log4j2.xml file within the Jira application installation directory
Was this helpful?