Jira Login fails after userCache being flushed
プラットフォームについて: Data Center のみ - この記事は、Data Center プラットフォームのアトラシアン製品にのみ適用されます。
この KB は Data Center バージョンの製品用に作成されています。Data Center 固有ではない機能の Data Center KB は、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
問題
For a Brief Period of around 5-10 mins, JIRA application login fails for all users after any bulk changes made to filters ownership or Dashboard ownership with ScriptRunner plugin installed
The following appears in the atlassian-jira.log in any one instance of the Data Center nodes.
****************************************************************************************
************************** removeAll called on a pinned cache **************************
****************************************************************************************
2019-02-01 10:18:31,292 http-nio-8080-exec-508 url:/rest/scriptrun...redEntityOwnership username:1578962 INFO 1578962 618x5255685x4 8x899z 10.141.73.80 /rest/scriptrunner/latest/canned/com.onresolve.scriptrunner.canned.jira.admin.ChangeSharedEntityOwnership [c.a.c.e.c.a.j.c.e.ofbiz.EagerOfBizUserCache.userCache] Cache com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache was flushed
2019-02-01 10:18:31,403 http-nio-8080-exec-387 url:/secure/QuickCreateIssue.jspa username:1600506 INFO 1600506 618x5255771x11 1luqkz 10.141.57.81 /secure/QuickCreateIssue.jspa [c.a.j.r.v2.issue.IssueBeanBuilder2] Cannot get value from RestAwareField customfield_16143, exception: 'comment should not be null!'
2019-02-01 10:18:31,647 http-nio-8080-exec-387 url:/secure/QuickCreateIssue.jspa username:1600506 INFO 1600506 618x5255771x11 1luqkz 10.141.57.81 /secure/QuickCreateIssue.jspa [c.a.j.r.v2.issue.IssueBeanBuilder2] Cannot get value from RestAwareField customfield_16143, exception: 'comment should not be null!'
2019-02-01 10:19:07,265 ehcache-replicator-8 WARN [c.a.j.cluster.distribution.JiraCacheManagerPeerProvider] Looking up rmiUrl //node1.example.com:40001/com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache threw a connection exception. This could mean that a node has gone offline or it may indicate network Example of stack trace from node updating the cache (note, this is from another incident):
"http-nio-8050-exec-141 url:/rest/scriptrun...redEntityOwnership username:user" #40195 daemon prio=5 os_prio=0 tid=0x00007fbdc407b800 nid=0x9f15 waiting on condition [0x00007fbebae9d000]
java.lang.Thread.State: WAITING (parking)
...
at com.atlassian.jira.cache.DeferredReplicationCache.put(DeferredReplicationCache.java:59)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$PutVisitor.visit(UserOrGroupCache.java:247)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$PutVisitor.visit(UserOrGroupCache.java:238)
at com.atlassian.jira.util.Functions$MappedVisitor.visit(Functions.java:198)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl$$Lambda$979/1369850049.accept(Unknown Source)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.forEach(SelectQueryImpl.java:231)
at com.atlassian.jira.entity.SelectQueryImpl$ExecutionContextImpl.visitWith(SelectQueryImpl.java:185)
at com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.visitAllUsingDatabase(EagerOfBizUserCache.java:84)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.buildCacheForced(UserOrGroupCache.java:149)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.buildCacheIfRequiredUnderLock(UserOrGroupCache.java:129)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.buildCacheIfRequired(UserOrGroupCache.java:118)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$1.create(UserOrGroupCache.java:42)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache$1.create(UserOrGroupCache.java:38)
at com.atlassian.util.concurrent.ResettableLazyReference$InternalReference.create(ResettableLazyReference.java:182)
at com.atlassian.util.concurrent.LazyReference$Sync.run(LazyReference.java:325)
at com.atlassian.util.concurrent.LazyReference.getInterruptibly(LazyReference.java:143)
at com.atlassian.util.concurrent.LazyReference.get(LazyReference.java:112)
at com.atlassian.util.concurrent.ResettableLazyReference.get(ResettableLazyReference.java:92)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.getCache(UserOrGroupCache.java:50)
at com.atlassian.jira.crowd.embedded.ofbiz.UserOrGroupCache.refresh(UserOrGroupCache.java:61)
at com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.flushCache(OfBizUserDao.java:986)
at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.flushCache(DelegatingUserDao.java:58)
at com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.flushCache(IndexedUserDao.java:473)
at com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.flushCache(DelegatingUserDao.java:58)
at com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.flushCache(SwitchingUserDao.java:30)
at com.atlassian.jira.crowd.embedded.ofbiz.OfBizCacheFlushingManager$OfBizCacheFlushingManagerListener.flushAllCaches(OfBizCacheFlushingManager.java:105)
at com.atlassian.jira.crowd.embedded.ofbiz.OfBizCacheFlushingManager$OfBizCacheFlushingManagerListener.onEvent(OfBizCacheFlushingManager.java:90)
...
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:177)
at com.onresolve.scriptrunner.canned.jira.admin.ChangeSharedEntityOwnership.doScript(ChangeSharedEntityOwnership.groovy:171)
...
at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.validateAndRunCanned(ScriptRunnerImpl.groovy:395)
at com.onresolve.scriptrunner.runner.rest.AbstractRestEndpoint.previewOrExecCanned(AbstractRestEndpoint.groovy:246)
at com.onresolve.scriptrunner.runner.rest.AbstractRestEndpoint.previewOrExecCanned(AbstractRestEndpoint.groovy:211)
at com.onresolve.scriptrunner.runner.rest.common.CannedScriptEndpoint.runCanned(CannedScriptEndpoint.groovy:125)
at com.onresolve.scriptrunner.runner.rest.common.CannedScriptEndpoint.runCannedFromJson(CannedScriptEndpoint.groovy:112)And in the Remaining Nodes, you would see the below errors:
2019-02-01 10:18:32,871 http-nio-8080-exec-499 url:/rest/gadget/1.0/login ERROR [o.a.c.c.C.[.[localhost].[/].[default]] Servlet.service() for servlet [default] in context with path [] threw exception
java.util.ConcurrentModificationException: User '1593511' no longer exists.
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.createLdapUserInLocalCache(DelegatedAuthenticationDirectory.java:328)
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.updateUserFromRemoteDirectory(DelegatedAuthenticationDirectory.java:277)
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.authenticateAndUpdateOrCreate(DelegatedAuthenticationDirectory.java:189)
at com.atlassian.crowd.directory.DelegatedAuthenticationDirectory.authenticate(DelegatedAuthenticationDirectory.java:158)
at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.authenticateUser(DirectoryManagerGeneric.java:311)
at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.authenticateUser(ApplicationServiceGeneric.java:198)
at com.atlassian.crowd.embedded.core.CrowdServiceImpl.authenticate(CrowdServiceImpl.java:75)
at com.atlassian.jira.security.login.JiraSeraphAuthenticator.crowdServiceAuthenticate(JiraSeraphAuthenticator.java:75)
at com.atlassian.jira.security.login.JiraSeraphAuthenticator.authenticate(JiraSeraphAuthenticator.java:49)
at com.atlassian.seraph.auth.DefaultAuthenticator.login(DefaultAuthenticator.java:88)
... 13 filtered
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)
... 19 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at com.valiantys.jira.plugins.sql.service.servletcontext.ContextListenerServletFilter.doFilter(ContextListenerServletFilter.java:24)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.ams.shipit.servlet.filter.AbstractThreadFilter.runWithChangedThreadName(AbstractThreadFilter.java:69)
at com.atlassian.ams.shipit.servlet.filter.AbstractThreadFilter.doFilter(AbstractThreadFilter.java:53)
at com.atlassian.ams.shipit.servlet.filter.AbstractThreadFilter.doFilter(AbstractThreadFilter.java:37)
... 3 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 26 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)診断
環境
Issue is expected to occur in Data center installation of JIRA along with ScriptRunner version 5.4.1+
原因
This is a known issue SRJIRA-2393 with ScriptRunner plugin version 5.1.6+ and resolved in 5.4.45 version.
As per the cause analysis
The Change Dashboard or Filter Ownership script can sometimes cause performance issues in very large instances running JIRA DataCenter. This is because we publish a ClearCacheEvent in order to make sure that, after the Dashboard and/or filters have changed ownership. While this doesn't typically cause problems, it can create issues when multiplied out across multiple nodes, since cache replication can take quite a bit of time.
回避策
Users should be able to login to the JIRA application again after the Usercache is rebuilt without any restart of the JIRA instances. But if the situation doesn't improve, then restarting JIRA would be the only work around.
ソリューション
Upgrading ScriptRunner to version 5.4.45+ should resolve this problem permanently. For more details please read through this: SRJIRA-2393
Last modified on Mar 21, 2024
Powered by Confluence and Scroll Viewport.