How to troubleshoot long page load times in Jira server

Summary

Jira page load times are taking more time than usual.

• This mainly happens after a Jira upgrade or after plugin upgrades.

• The pages show loading transactions that take seconds to finish. This is especially noticeable when loading batch.js and batch.css files.

• The issue gets worse if we use browser incognito mode or if we disable browser caching.

• The load times are showing as below if we generate a har file from the browser:

Diagnosis

This KB applies when

• Jira was upgraded recently or Jira plugins were updated recently.

• Jira runs under load, with many concurrent users 100+.

• Jira has many plugins injecting code in batch.js/batch.css

• Jira runs on default Java CodeCache value - 240MB

• Jira runs behind a reverse proxy on a high-latency network.

Diagnostic Steps

• Need to generate HAR files for the slow pages in Jira, we need to check the loading times of batch.js and batch.css

• Verify the loading times by checking Jira Tomcat access logs.

• Need to collect thread dumps and thread CPU information on the impacted instance.

• Need to check the Tomcat console for any related CodeCache messages.

• Need to verify if the customer browser has caching enabled or not.

Cause

There could be multiple causes that contribute to this problem:

Cause #1: Browser caching disabled/Slow network.

• Jira generates Javascript and Styles on the fly based on the user language, project permissions, and plugins being used.

• The files can become large, sometimes reaching more than 7MB compressed. Jira has gzip compression enabled by default to help make the network transfer fast.

• Jira also caches the files on the server side to help speed up processing, but the cache could expire if the user properties change quickly, e.g., the user changes his language, or changes his project permissions.

• Also, plugin update/installation flushes the Jira server-side cache, which could make things slower.

• Jira will also rely on browsing to cache those resources on the client side, as it will take time to transfer those files on the network, especially if the network has high latency.

• Caching the files on the browser will have those files loaded from the client disk, and will save the time of the network transfer.

• The browser should get a 304 not modified (loaded from disk) on checking if the file is still the same, this means that Jira still has the same version of the file as the file cached by the browser; this will save the network time.

• If the file is changed, the browser should stream it from Jira, and you should see a full transfer with a normal HTTP 200 signaling success.

• If the file needs to be transferred every time a request is made on a slow network (no browser caching), it will cause significant delay, Jira will use multiple batch.js and batch.css files on the same page depending on the number of plugins and the page structure missing browser cache on a slow network will lead to very long network delays.

Cause#2: Jira is running out of CodeCache

• The Jira node will show very high CPU usage and the whole Jira instance will be very slow.

• Looking at Jira thread dumps, you will see that the Java Compiler threads are runnable all the time:

  1 2 3 4 5 "C1 CompilerThread10" #39 daemon prio=9 os_prio=0 tid=0x00007f66a9c59000 nid=0x58d8 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread9" #38 daemon prio=9 os_prio=0 tid=0x00007f66a9c56800 nid=0x58d7 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

• The Compiler threads will be using most of the available CPU, starving other threads for longer times, specifically, we will also see many threads trying to render statics as long-running threads, having below stack traces:

• One other contributing factor will be the loss of optimized code for the Mozilla Rhino compiler library from CodeCache, this code renders the statics for each request - if they are not cached on the server. If the optimized Rhino code is not in the CodeCache, it will run extremely slowly and will cause further slowness. This becomes a big bottleneck, especially for very busy Jira instances, as the Mozilla Rhino compiler is used very frequently, which meets one of the JIT compilation criteria and is supposed to be optimized by the JVM.

Cause#3: Jira's JIT Compiler got disabled due to full CodeCache, this is related to Cause#2.

• In this case, CodeCache would be overwhelmed and filled up; the Compiler threads would not be able to store any more code in the CodeCache, and the compiler would get disabled.

• We will see similar logs in the Tomcat console logs:

  1 2 3 4 5 6 Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= CodeCache: size=245760Kb used=244153Kb max_used=244189Kb free=1606Kb bounds [0x00007f2709000000, 0x00007f2718000000, 0x00007f2718000000] total_blobs=45381 nmethods=43794 adapters=1490 compilation: disabled (not enough contiguous free space left)

• Looking into Jira thread dumps will not show any Compiler threads since the compiler was disabled.

• Jira will not run any optimized code; it will use standard JVM-interpreted byte code for all operations, which will cause a huge performance hit.

• In this case, we might not notice much CPU usage since we have no compiler threads running.

• Once more, we will also see long-running Mozilla Rhino compiler threads in the thread dumps.

Cause#4: Incompatible custom HTML/CSS/Javascript is being injected into the page.

• The issue persists after disabling installed apps by entering Safe Mode.

• Disabling the HTML markup support "Enable HTML in ..." options within ⚙️ →System → General configuration resolves the issue.

• In this case, there is custom HTML/CSS/Javascript being injected into the browser from either the customfield, fieldconfiguration, fieldconfigscheme,fieldlayoutitem, or the Announcement Banner that is no longer compatible with the application.

• You can verify this by reviewing the entries in these tables: How to identify fields with custom Javascript in their description in Jira Data Center / Server

Solution

• Cause#1:

  • We need to make sure that browser caching is enabled all the time.

  • We need to make sure that Gzip compression is enabled ether on JIRA or better on the reserve proxy, this should save JIRA from losing CPU cycles on compression.

  • We might want to introduce an extra caching layer by using a caching office proxy: eg: Squid, this should help keep the large statics cached close to the client if the connection to Jira has high latency.

  • In case Jira runs behind a reverse proxy, we might want to enable mod_expires , this could help with setting content expiration times, please check this KB for more details: Jira is Slow Due to High Latency Connections While Using Reverse Proxy.

  • Make sure that the client machine has enough resources from CPU and memory perspective to be able to work with compressed content.

  • Need to avoid doing any plugin updates or installations during Jira usage peak hours.

• Cause#2 & Cause#3:

  • We need to increase CodeCache by adding the below start up option to Jira.

    1 -XX:ReservedCodeCacheSize=512m

• Cause#4

  • We need to remove the offending code from either the UI at the relevant admin page (or directly from the database).

  • Always back up your data before making any database modifications. If possible, test any alter, insert, update, or delete SQL commands on a staging server first.

    Use the queries from How to identify fields with custom Javascript in their description in Jira Data Center to identify the fields.