Full Reindex failing with Out Of Memory due to the size of cache net.sf.ehcache.store.chm.SelectableConcurrentHashMap

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

要約

Full Reindex failing due to OutofMemory caused by SLA timeline calculation.

環境

4.20.13
.

診断

  • Enable all the required Java arguments to generate a Heapdump when an Out of Memory occurs. Reference: Analyze OutofMemory errors in Jira server with Heap Dumps and reproduce the problem, running a full reindex;
  • Using a tool to analyze the Heapdump, we could observe a big amount of data related to the SLA timeline. They are loaded into the cache net.sf.ehcache.store.chm.SelectableConcurrentHashMap:

  • A piece of the SLA timeline observed on this cache:
{"timeline":{"events":[{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},
{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]},{"date":1660213800000,"types":["START"]}


  • Stack Trace:
Thread Stack 
IssueIndexer:thread-11
  at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I (Native Method)
  at java.net.SocketInputStream.socketRead(Ljava/io/FileDescriptor;[BIII)I (SocketInputStream.java:116)
  at java.net.SocketInputStream.read([BIII)I (SocketInputStream.java:171)
  at java.net.SocketInputStream.read([BII)I (SocketInputStream.java:141)
  at com.mysql.jdbc.util.ReadAheadInputStream.fill(I)V (ReadAheadInputStream.java:101)
  at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary([BII)I (ReadAheadInputStream.java:144)
  at com.mysql.jdbc.util.ReadAheadInputStream.read([BII)I (ReadAheadInputStream.java:174)
  at com.mysql.jdbc.MysqlIO.readFully(Ljava/io/InputStream;[BII)I (MysqlIO.java:2966)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3427)
  at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3417)
  at com.mysql.jdbc.MysqlIO.checkErrorPacket(I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3860)
  at com.mysql.jdbc.MysqlIO.sendCommand(ILjava/lang/String;Lcom/mysql/jdbc/Buffer;ZLjava/lang/String;I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:2524)
  at com.mysql.jdbc.MysqlIO.sqlQueryDirect(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;Ljava/lang/String;Lcom/mysql/jdbc/Buffer;IIIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods; (MysqlIO.java:2675)
  at com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (ConnectionImpl.java:2465)
  at com.mysql.jdbc.PreparedStatement.executeInternal(ILcom/mysql/jdbc/Buffer;ZZ[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (PreparedStatement.java:1915)
  at com.mysql.jdbc.PreparedStatement.executeQuery()Ljava/sql/ResultSet; (PreparedStatement.java:2023)
  at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
  at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
  at com.atlassian.jira.ofbiz.sql.PreparedStatementWrapper.executeQuery()Ljava/sql/ResultSet; (PreparedStatementWrapper.java:42)
  at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.lambda$executeQuery$5()Ljava/sql/ResultSet; (DiagnosticPreparedStatement.java:59)
  at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement$$Lambda$2806.execute()Ljava/lang/Object; (Unknown Source)
  at com.atlassian.diagnostics.internal.platform.monitor.db.DefaultDatabaseDiagnosticsCollector.recordExecutionTime(Lcom/atlassian/diagnostics/internal/platform/monitor/db/SqlOperation;Ljava/lang/String;)Ljava/lang/Object; (DefaultDatabaseDiagnosticsCollector.java:70)
  at com.atlassian.jira.diagnostic.connection.DatabaseDiagnosticsCollectorDelegate.recordExecutionTime(Lcom/atlassian/diagnostics/internal/platform/monitor/db/SqlOperation;Ljava/lang/String;)Ljava/lang/Object; (DatabaseDiagnosticsCollectorDelegate.java:55)
  at com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DiagnosticPreparedStatement.java:59)
  at com.querydsl.sql.AbstractSQLQuery.iterateSingle(Lcom/querydsl/core/QueryMetadata;Lcom/querydsl/core/types/Expression;)Lcom/mysema/commons/lang/CloseableIterator; (AbstractSQLQuery.java:370)
  at com.querydsl.sql.AbstractSQLQuery.iterate()Lcom/mysema/commons/lang/CloseableIterator; (AbstractSQLQuery.java:342)
  at com.querydsl.core.group.GroupByMap.transform(Lcom/querydsl/core/FetchableQuery;)Ljava/util/Map; (GroupByMap.java:54)
  at com.querydsl.core.group.GroupByMap.transform(Lcom/querydsl/core/FetchableQuery;)Ljava/lang/Object; (GroupByMap.java:35)
  at com.querydsl.core.support.FetchableQueryBase.transform(Lcom/querydsl/core/ResultTransformer;)Ljava/lang/Object; (FetchableQueryBase.java:55)
  at com.atlassian.servicedesk.internal.sla.customfield.JIRACustomFieldValueStore.lambda$getAllSLATextValues$0(Ljava/lang/Long;Lcom/atlassian/pocketknife/api/querydsl/DatabaseConnection;)Ljava/util/Map; (JIRACustomFieldValueStore.java:42)
  at com.atlassian.servicedesk.internal.sla.customfield.JIRACustomFieldValueStore$$Lambda$3208.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.lambda$execute$0(Ljava/util/function/Function;Ljava/sql/Connection;)Ljava/lang/Object; (DatabaseAccessorImpl.java:69)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl$$Lambda$3179.execute(Ljava/sql/Connection;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.executeInternal(Ljava/sql/Connection;Lcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (DefaultTransactionalExecutor.java:87)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.lambda$execute$0(Lcom/atlassian/sal/api/rdbms/ConnectionCallback;Ljava/sql/Connection;)Ljava/lang/Object; (DefaultTransactionalExecutor.java:43)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor$$Lambda$1289.execute(Ljava/sql/Connection;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor$$Lambda$1290.apply(Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.jira.database.DatabaseAccessorImpl.runInManagedTransaction(Ljava/util/function/Function;)Ljava/lang/Object; (DatabaseAccessorImpl.java:131)
  at sun.reflect.GeneratedMethodAccessor138.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:498)
  at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (ContextClassLoaderSettingInvocationHandler.java:26)
  at com.sun.proxy.$Proxy313.runInManagedTransaction(Ljava/util/function/Function;)Ljava/lang/Object; (Unknown Source)
  at sun.reflect.GeneratedMethodAccessor138.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:498)
  at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object; (HostComponentFactoryBean.java:130)
  at com.sun.proxy.$Proxy313.runInManagedTransaction(Ljava/util/function/Function;)Ljava/lang/Object; (Unknown Source)
  at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.runInStartedOrExistingTransaction(Lcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (JiraHostConnectionAccessor.java:130)
  at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.execute(ZZLcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (JiraHostConnectionAccessor.java:60)
  at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.execute(Lcom/atlassian/sal/api/rdbms/ConnectionCallback;)Ljava/lang/Object; (DefaultTransactionalExecutor.java:40)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.execute(Ljava/util/function/Function;ZZLcom/atlassian/pocketknife/api/querydsl/util/OnRollback;)Ljava/lang/Object; (DatabaseAccessorImpl.java:67)
  at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.runInTransaction(Ljava/util/function/Function;Lcom/atlassian/pocketknife/api/querydsl/util/OnRollback;)Ljava/lang/Object; (DatabaseAccessorImpl.java:43)
  at com.atlassian.servicedesk.internal.sla.customfield.JIRACustomFieldValueStore.getAllSLATextValues(Ljava/lang/Long;)


  • SQL statistics collected by the DBA while running a reindex are showing high response time for SLA table:
ランクQuery ID応答時間%CallsR/CallV/M項目
10x8845FDA210EBA1B41693978.716338.2%5922861.450587...SELECT UNION AO_?E_SLAAUDITLOG


  • During the reindex, we can observe warnings about issues that will not have the SLA timeline indexed because they are marked as too big:
2022-11-14 09:40:50,189-0800 IssueIndexer:thread-2 WARN xxxx /secure/admin/IndexReIndex!reindex.jspa [c.a.s.i.sla.searcher.SLACustomFieldIndexer] Not indexing timeline for xxxx-99999, because it is too big.


原因

These SLA timelines are big due to the SLA configuration, causing the issues to have a long history of start/stop/pause events. 
Although the cache is configured to automatically clear items out after 10 minutes,  the EHCache documentation says "elements can stay in the cache after expiration and actually expired elements are only evicted when it is accessed or resource constraints are reached".

We've run into customers experiencing out-of-memory issues previously as we used to allow an unlimited size for the cache, however, that was resolved as part of the ticket JSDSERVER-5468 - Getting issue details... STATUS .

A default maximum value of 2 million items (where items are the number of issues) was added for nodes in cluster environments, and it was made configurable by the system property servicedesk.slavalue.global.cache.size.limit. This value can be configured to a smaller value, for instance, 250k, reducing the number of issues loaded into the cache.


ソリューション

Edit the jira-config.properties file and add the system property into the jira-config.properties file (Reference: Edit the jira-config.properties file in Jira server)

  • Shut down your JIRA applications;
  • Open the jira-config.properties file (located at the root of your JIRA application home directory) in a text editor;

 新しいインストールや、以前の Jira アプリケーション バージョンのカスタマイズを行ったことがないアップグレード済みのインストールの場合、このファイルが存在しないことがあります。このファイルが存在しない場合、テキスト エディタを使用してファイルを作成します。

  • Add the property servicedesk.slavalue.global.cache.size.limit=250000 in this file;
  • Save your modifications to the jira-config.properties file;
  • Jira アプリケーションを再起動します。


最終更新日: 2023 年 1 月 3 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.