Jira Login fails after userCache being flushed
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
For a Brief Period of around 5-10 mins, JIRA application login fails for all users after any bulk changes made to filters ownership or Dashboard ownership with ScriptRunner plugin installed
The following appears in the atlassian-jira.log in any one instance of the Data Center nodes.
1
2
3
4
5
6
7
****************************************************************************************
************************** removeAll called on a pinned cache **************************
****************************************************************************************
2019-02-01 10:18:31,292 http-nio-8080-exec-508 url:/rest/scriptrun...redEntityOwnership username:1578962 INFO 1578962 618x5255685x4 8x899z 10.141.73.80 /rest/scriptrunner/latest/canned/com.onresolve.scriptrunner.canned.jira.admin.ChangeSharedEntityOwnership [c.a.c.e.c.a.j.c.e.ofbiz.EagerOfBizUserCache.userCache] Cache com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache was flushed
2019-02-01 10:18:31,403 http-nio-8080-exec-387 url:/secure/QuickCreateIssue.jspa username:1600506 INFO 1600506 618x5255771x11 1luqkz 10.141.57.81 /secure/QuickCreateIssue.jspa [c.a.j.r.v2.issue.IssueBeanBuilder2] Cannot get value from RestAwareField customfield_16143, exception: 'comment should not be null!'
2019-02-01 10:18:31,647 http-nio-8080-exec-387 url:/secure/QuickCreateIssue.jspa username:1600506 INFO 1600506 618x5255771x11 1luqkz 10.141.57.81 /secure/QuickCreateIssue.jspa [c.a.j.r.v2.issue.IssueBeanBuilder2] Cannot get value from RestAwareField customfield_16143, exception: 'comment should not be null!'
2019-02-01 10:19:07,265 ehcache-replicator-8 WARN [c.a.j.cluster.distribution.JiraCacheManagerPeerProvider] Looking up rmiUrl //node1.example.com:40001/com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache threw a connection exception. This could mean that a node has gone offline or it may indicate network Example of stack trace from node updating the cache (note, this is from another incident):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
"http-nio-8050-exec-141 url:/rest/scriptrun...redEntityOwnership username:user" #40195 daemon prio=5 os_prio=0 tid=0x00007fbdc407b800 nid=0x9f15 waiting on condition [0x00007fbebae9d000]
java.lang.Thread.State: WAITING (parking)
...
at com.atlassian.jira.cache.DeferredReplicationCache.put(DeferredReplicationCache.java:59)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$PutVisitor.visit(UserOrGroupCache.java:247)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$PutVisitor.visit(UserOrGroupCache.java:238)
at com.atlassian.jira.util.Functions$MappedVisitor.visit(Functions.java:198)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl$$Lambda$979/1369850049.accept(Unknown Source)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:231)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.visitWith(SelectQueryImpl.java:185)
at com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.visitAllUsingDatabase(EagerOfBizUserCache.java:84)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.buildCacheForced(UserOrGroupCache.java:149)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.buildCacheIfRequiredUnderLock(UserOrGroupCache.java:129)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.buildCacheIfRequired(UserOrGroupCache.java:118)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$1.create(UserOrGroupCache.java:42)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$1.create(UserOrGroupCache.java:38)
at com.atlassian.util.concurrent.ResettableLazyReference$InternalReference.create(ResettableLazyReference.java:182)
at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
at com.atlassian.util.concurrent.ResettableLazyReference.get(ResettableLazyReference.java:92)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.getCache(UserOrGroupCache.java:50)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.refresh(UserOrGroupCache.java:61)
at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.flushCache(OfBizUserDao.java:986)
at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.flushCache(DelegatingUserDao.java:58)
at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.flushCache(IndexedUserDao.java:473)
at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.flushCache(DelegatingUserDao.java:58)
at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.flushCache(SwitchingUserDao.java:30)
at com.atlassian.jira.crowd.embedded.ofbiz.OfBizCacheFlushingManager$OfBizCacheFlushingManagerListener.flushAllCaches(OfBizCacheFlushingManager.java:105)
at com.atlassian.jira.crowd.embedded.ofbiz.OfBizCacheFlushingManager$OfBizCacheFlushingManagerListener.onEvent(OfBizCacheFlushingManager.java:90)
...
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:177)
at com.onresolve.scriptrunner.canned.jira.admin.ChangeSharedEntityOwnership.doScript(ChangeSharedEntityOwnership.groovy:171)
...
at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.validateAndRunCanned(ScriptRunnerImpl.groovy:395)
at com.onresolve.scriptrunner.runner.rest.AbstractRestEndpoint.previewOrExecCanned(AbstractRestEndpoint.groovy:246)
at com.onresolve.scriptrunner.runner.rest.AbstractRestEndpoint.previewOrExecCanned(AbstractRestEndpoint.groovy:211)
at com.onresolve.scriptrunner.runner.rest.common.CannedScriptEndpoint.runCanned(CannedScriptEndpoint.groovy:125)
at com.onresolve.scriptrunner.runner.rest.common.CannedScriptEndpoint.runCannedFromJson(CannedScriptEndpoint.groovy:112)And in the Remaining Nodes, you would see the below errors:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
2019-02-01 10:18:32,871 http-nio-8080-exec-499 url:/rest/gadget/1.0/login ERROR [o.a.c.c.C.[.[localhost].[/].[default]] Servlet.service() for servlet [default] in context with path [] threw exception
java.util.ConcurrentModificationException: User '1593511' no longer exists.
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.createLdapUserInLocalCache(DelegatedAuthenticationDirectory.java:328)
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.updateUserFromRemoteDirectory(DelegatedAuthenticationDirectory.java:277)
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.authenticateAndUpdateOrCreate(DelegatedAuthenticationDirectory.java:189)
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.authenticate(DelegatedAuthenticationDirectory.java:158)
at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:311)
at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:198)
at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:75)
at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)
at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)
at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:88)
... 13 filtered
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)
... 19 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at com.valiantys.jira.plugins.sql.service.servletcontext.ContextListenerServletFilter.doFilter(ContextListenerServletFilter.java:24)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.ams.shipit.servlet.filter.AbstractThreadFilter.runWithChangedThreadName(AbstractThreadFilter.java:69)
at com.atlassian.ams.shipit.servlet.filter.AbstractThreadFilter.doFilter(AbstractThreadFilter.java:53)
at com.atlassian.ams.shipit.servlet.filter.AbstractThreadFilter.doFilter(AbstractThreadFilter.java:37)
... 3 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 26 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)Diagnosis
Environment
Issue is expected to occur in Data center installation of JIRA along with ScriptRunner version 5.4.1+
Cause
This is a known issue SRJIRA-2393 with ScriptRunner plugin version 5.1.6+ and resolved in 5.4.45 version.
As per the cause analysis
The Change Dashboard or Filter Ownership script can sometimes cause performance issues in very large instances running JIRA DataCenter. This is because we publish a ClearCacheEvent in order to make sure that, after the Dashboard and/or filters have changed ownership. While this doesn't typically cause problems, it can create issues when multiplied out across multiple nodes, since cache replication can take quite a bit of time.
Solution
Workaround
Users should be able to login to the JIRA application again after the Usercache is rebuilt without any restart of the JIRA instances. But if the situation doesn't improve, then restarting JIRA would be the only work around.
Resolution
Upgrading ScriptRunner to version 5.4.45+ should resolve this problem permanently. For more details please read through this: SRJIRA-2393
Updated on April 2, 2025
Was this helpful?
Still need help?
The Atlassian Community is here for you.