High memory usage due to leaking ThreadLocal variables from GroovyCustomField
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
At large JIRA instances and heavy load GroovyCustomField can leak ThreadLocal variables, that will cause high memory usage and lead to OOM.
Diagnosis
Environment
ScriptRunner for JIRA version 4.0 - 5.0.14
Diagnostic Steps
From heap dump you can see the following problem:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Class Name | Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------
https-jsse-nio-10.35.136.254-8443-exec-115 Thread
org.apache.tomcat.util.threads.TaskThread @ 0x35d8687b8 | 128 | 110,127,096 | 0.61%
|- java.lang.ThreadLocal$ThreadLocalMap @ 0x6213b4820 | 24 | 110,121,168 | 0.61%
| '- java.lang.ThreadLocal$ThreadLocalMap$Entry[32768] @ 0x360a14568 | 131,088 | 110,121,144 | 0.61%
| |- java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x5937be410 | 32 | 109,711,184 | 0.61%
| | '- java.util.LinkedHashMap @ 0x5937be430 | 56 | 109,711,152 | 0.61%
| | |- java.util.HashMap$Node[524288] @ 0x38082b4a8 | 2,097,168 | 2,097,168 | 0.01%
| | |- java.util.LinkedHashMap$Entry @ 0x4b3af5040 | 40 | 298,720 | 0.00%
| | |- java.util.LinkedHashMap$Entry @ 0x393ac05e8 | 40 | 216,392 | 0.00%
....
| | |- java.util.LinkedHashMap$Entry @ 0x35d057238 | 40 | 97,344 | 0.00%
| | '- Total: 25 of 247,676 entries; 247,651 more | | | Note 247k entries for ThreadLocal, and size 100MB. Referent: com.onresolve.scriptrunner.customfield.GroovyCustomField$1
Histogram:
Class Name | Objects | Shallow Heap | Retained Heap |
|---|---|---|---|
char[] | 70,599,174 | 9,251,242,216 | >= 9,251,242,216 |
java.lang.String | 70,488,632 | 1,691,727,168 | >= 10,844,317,840 |
java.util.LinkedHashMap$Entry | 20,774,537 | 830,981,480 | >= 8,357,527,664 |
java.util.HashMap$Node | 18,619,919 | 595,837,408 | >= 4,209,784,280 |
java.util.HashMap$Node[] | 2,300,341 | 383,801,528 | >= 4,551,997,344 |
java.util.HashMap | 2,230,648 | 107,071,104 | >= 4,449,251,032 |
So we see LinkedHashMap using 8GB+ data and there are 20M objecst, same structure as above.
If we check specifically ThreadLocal, we can see that they are using 8GB+
1
2
3
4
5
6
7
8
9
Class Name | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------
java.lang.ThreadLocal$ThreadLocalMap | 870 | 20,880 | >= 8,695,572,600
|- java.lang.ThreadLocal$ThreadLocalMap$Entry[] | 870 | 8,828,768 |
| |- java.lang.Class | 1 | 0 |
| |- java.lang.ThreadLocal$ThreadLocalMap$Entry | 206,026 | 6,592,832 |
| | |- char[] | 468 | 632,624 | >= 632,624
| | |- com.atlassian.jira.web.filters.ThreadLocalQueryProfiler| 302 | 7,248 | >= 8,916,904
| | |- java.util.LinkedHashMap | 161 | 9,016 | >= 8,516,389,472Cause
Bug in ScriptRunner:
The problem is that script field values are cached in thread locals, when there is a full reindex done for instance, there will be a long-running thread, and the script field values are not cleared from the thread local.
see related;
https://productsupport.adaptavist.com/browse/SRJIRA-2570
https://productsupport.adaptavist.com/browse/SRJIRA-551
Solution
Resolution
Upgrade ScriptRunner to 5.2.2+ version. Note it's compatible with Jira Server 7.2.0+.
Was this helpful?