Confluence intermittent performance degradation and slow page loads due to SentinelOne Agent
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Confluence may suffer temporary and intermittent periods of performance degradation specifically impacting page loads (i.e. requests for /pages/viewpage.action). The exact length of these periods and their frequency may vary.
Environment
8.5.5, 8.6.1
Diagnosis
Thread dumps captured during the performance event reveals HTTP threads with a stack trace similar to:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
14:47:35 - http-nio-8090-exec-23 url: /pages/viewpage.action; user: LouPag
State:RUNNABLE
CPU usage:0.00%
Running for: 0:00.00
Waiting for
This thread is not waiting for notification on any lock
Locks held
This thread does not hold any locks
Stack trace
org.springframework.beans.BeanWrapperImpl.getLocalPropertyHandler(BeanWrapperImpl.java:230)
org.springframework.beans.BeanWrapperImpl.getLocalPropertyHandler(BeanWrapperImpl.java:63)
org.springframework.beans.AbstractNestablePropertyAccessor.getPropertyHandler(AbstractNestablePropertyAccessor.java:730)
org.springframework.beans.AbstractNestablePropertyAccessor.isWritableProperty(AbstractNestablePropertyAccessor.java:567)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1709)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireBeanProperties(AbstractAutowireCapableBeanFactory.java:417)
com.atlassian.spring.container.SpringContainerContext.autowireComponent(SpringContainerContext.java:102)
com.atlassian.spring.container.ContainerManager.autowireComponent(ContainerManager.java:58)
com.atlassian.confluence.util.GeneralUtil.newWiredConfluenceActionSupport(GeneralUtil.java:180)
com.atlassian.confluence.renderer.radeox.macros.MacroUtils.getConfluenceActionSupport(MacroUtils.java:88)
com.atlassian.confluence.renderer.radeox.macros.MacroUtils.createDefaultVelocityContext(MacroUtils.java:75)
com.atlassian.confluence.renderer.radeox.macros.MacroUtils.defaultVelocityContext(MacroUtils.java:43)
com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebFragmentHelper.getContext(ConfluenceWebFragmentHelper.java:79)
com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebFragmentHelper.renderVelocityFragment(ConfluenceWebFragmentHelper.java:72)
com.atlassian.plugin.web.model.DefaultWebLink.getRenderedUrl(DefaultWebLink.java:32)
com.atlassian.plugin.web.model.DefaultWebLink.getDisplayableUrl(DefaultWebLink.java:40)
com.atlassian.confluence.plugin.descriptor.web.model.ConfluenceWebLink.getDisplayableUrl(ConfluenceWebLink.java:80)
com.atlassian.confluence.plugin.descriptor.web.model.ConfluenceWebLink.getDisplayableUrl(ConfluenceWebLink.java:70)
jdk.internal.reflect.GeneratedMethodAccessor987.invoke(Unknown Source)
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.8.1/Unknown Source)
java.lang.reflect.Method.invoke(java.base@17.0.8.1/Unknown Source)Additionally, a review of stuck thread log warnings found within the Tomcat catalina log files reveal a build-up of HTTP threads reporting stuck with a similar stack trace. In this context, stuck refers to threads flagged for running for longer than the 60 second (default) threshold defined in the server.xml for the StuckThreadDetectionValve.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
26-Jan-2024 09:11:10.831 WARNING [Catalina-utility-2] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-2 url: /pages/viewpage.action; user: LouPag] (id=[358]) has been active for [61,933] milliseconds (since [1/26/24 9:10 AM]) to serve the same request for [<base-URL>/pages/viewpage.action?pageId=99999999999] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [16] thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at org.springframework.beans.BeanWrapperImpl.getLocalPropertyHandler(BeanWrapperImpl.java:230)
at org.springframework.beans.BeanWrapperImpl.getLocalPropertyHandler(BeanWrapperImpl.java:63)
at org.springframework.beans.AbstractNestablePropertyAccessor.getPropertyHandler(AbstractNestablePropertyAccessor.java:730)
at org.springframework.beans.AbstractNestablePropertyAccessor.isWritableProperty(AbstractNestablePropertyAccessor.java:567)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1709)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1452)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireBeanProperties(AbstractAutowireCapableBeanFactory.java:417)
at com.atlassian.spring.container.SpringContainerContext.autowireComponent(SpringContainerContext.java:102)
at com.atlassian.spring.container.ContainerManager.autowireComponent(ContainerManager.java:58)
at com.atlassian.confluence.util.GeneralUtil.newWiredConfluenceActionSupport(GeneralUtil.java:180)
at com.atlassian.confluence.renderer.radeox.macros.MacroUtils.getConfluenceActionSupport(MacroUtils.java:88)
at com.atlassian.confluence.renderer.radeox.macros.MacroUtils.createDefaultVelocityContext(MacroUtils.java:75)
at com.atlassian.confluence.renderer.radeox.macros.MacroUtils.defaultVelocityContext(MacroUtils.java:43)
at com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebFragmentHelper.getContext(ConfluenceWebFragmentHelper.java:79)
at com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebFragmentHelper.renderVelocityFragment(ConfluenceWebFragmentHelper.java:72)
at com.atlassian.plugin.web.model.DefaultWebLink.getRenderedUrl(DefaultWebLink.java:32)
at com.atlassian.plugin.web.model.DefaultWebLink.getDisplayableUrl(DefaultWebLink.java:40)
at com.atlassian.confluence.plugin.descriptor.web.model.ConfluenceWebLink.getDisplayableUrl(ConfluenceWebLink.java:80)
at com.atlassian.confluence.spaces.actions.SpaceToolsTabsHelper$SpaceToolsTabs.addTopLevelTab(SpaceToolsTabsHelper.java:210)
at com.atlassian.confluence.spaces.actions.SpaceToolsTabsHelper.lambda$getSpaceToolsTabs$3(SpaceToolsTabsHelper.java:146)
at com.atlassian.confluence.spaces.actions.SpaceToolsTabsHelper$$Lambda$5739/0x0000000084a8fd60.accept(Unknown Source)
at java.base@17.0.8.1/java.lang.Iterable.forEach(Unknown Source)
at com.atlassian.confluence.spaces.actions.SpaceToolsTabsHelper.getSpaceToolsTabs(SpaceToolsTabsHelper.java:145)
at com.atlassian.confluence.spaces.actions.SpaceToolsTabsHelper.getNavigation(SpaceToolsTabsHelper.java:102)
at com.atlassian.confluence.spaces.actions.SpaceToolsTabsHelper.getNavigation(SpaceToolsTabsHelper.java:79)
at jdk.internal.reflect.GeneratedMethodAccessor1914.invoke(Unknown Source)
at java.base@17.0.8.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base@17.0.8.1/java.lang.reflect.Method.invoke(Unknown Source)
...........
...
.While the full stack trace from your environment may not match exactly the above two snippets, examples observed across cases are very similar to the above.
Review of the loaded Java parameters indicate that the SentinelOne agent is in use.
1
-agentpath:C:\Program Files\SentinelOne\Sentinel Agent 22.3.4.612\SentinelJava64.dllCause
The SentinelOne agent is responsible for this performance issue.
Solution
As mentioned in the Performance Tuning documentation:
Antivirus software greatly decreases the performance of Confluence. Antivirus software that intercepts access to the hard disk is particularly detrimental, and may even cause errors with Confluence. You should configure your antivirus software to ignore the Confluence home directory, its index directory and any database-related directories.
With that noted, this issue can be prevented by adding exclusions to the tomcat9w.exe process on Windows hosts, or otherwise adding exclusions to the Confluence Install and Home directories.
Important
Before taking any action such as disabling or adding exclusions, first check with your local security team or security practitioner. You can also consider contacting the vendor SentinelOne for more additional support .
⚠️ The -agentpath parameter may be injected by SentinelOne via a non-standard and presently unknown method outside the usual methods documented - Configuring System Properties. Therefore, it might be necessary to first add exclusions, then fully restart Confluence for the change to be picked up.
Was this helpful?