Re-index on Jira fails due to an OutOfMemoryError when connected to a MySQL database
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
問題
When attempting to perform a re-index on the instance connected to a MySQL database, it fails due to an OutOfMemoryError, even if the heap size is adequated based on our Jira Sizing Guide. The following appears in the atlassian-jira.log:
2019-01-07 12:49:13,367 JiraTaskExectionThread-1 WARN admin@admin.com 763x101x1 1plfa7f 192.168.1.1 /secure/admin/IndexReIndex.jspa [c.a.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '1897157'
2019-01-07 12:49:13,367 JiraTaskExectionThread-1 WARN admin@admin.com 763x101x1 1plfa7f 192.168.1.1 /secure/admin/IndexReIndex.jspa [c.a.jira.index.AccumulatingResultBuilder] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.atlassian.jira.index.FutureResult.await(FutureResult.java:29)
at com.atlassian.jira.index.AccumulatingResultBuilder.collectResult(AccumulatingResultBuilder.java:93)
at com.atlassian.jira.index.AccumulatingResultBuilder.addInternal(AccumulatingResultBuilder.java:58)
at com.atlassian.jira.index.AccumulatingResultBuilder.add(AccumulatingResultBuilder.java:35)
at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:1014)
at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:991)
at com.atlassian.jira.issue.index.DefaultIndexManager.lambda$reIndexAll$0(DefaultIndexManager.java:328)
at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:377)This issue has not been observed yet on different database types.
診断
A detailed analysis of a heap dump prior to the crash illustrates that the top consumers on the dominator tree are related to indexing threads, as shown on example below:
Class Name | Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x67ec03900 IssueIndexer:thread-7 Thread | 120 | 1,862,908,776 | 29.55%
java.lang.Thread @ 0x67ec0a278 IssueIndexer:thread-6 Thread | 120 | 1,000,382,352 | 15.87%
java.lang.Thread @ 0x67ec0c530 IssueIndexer:thread-3 Thread | 120 | 800,610,288 | 12.70%
java.lang.Thread @ 0x67ec08270 IssueIndexer:thread-9 Thread | 120 | 585,851,264 | 9.29%
java.lang.Thread @ 0x67ec01328 IssueIndexer:thread-1 Thread | 120 | 585,616,720 | 9.29%
java.lang.Thread @ 0x67ec05dc8 IssueIndexer:thread-10 Thread| 120 | 322,734,384 | 5.12%
java.lang.Thread @ 0x67ea705c0 IssueIndexer:thread-8 Thread | 120 | 322,068,320 | 5.11%
java.lang.Thread @ 0x67ebfeec0 IssueIndexer:thread-4 Thread | 120 | 136,338,712 | 2.16%
java.lang.Thread @ 0x67e885448 IssueIndexer:thread-5 Thread | 120 | 101,507,576 | 1.61%
java.lang.Thread @ 0x67ebfc968 IssueIndexer:thread-2 Thread | 120 | 100,766,272 | 1.60%By checking the equivalent thread dump for the top consumer object we can see the following stack trace:
IssueIndexer:thread-7
at java.net.SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I (Native Method)
at java.net.SocketInputStream.socketRead(Ljava/io/FileDescriptor;[BIII)I (SocketInputStream.java:116)
at java.net.SocketInputStream.read([BIII)I (SocketInputStream.java:171)
at java.net.SocketInputStream.read([BII)I (SocketInputStream.java:141)
at com.mysql.jdbc.util.ReadAheadInputStream.fill(I)V (ReadAheadInputStream.java:101)
at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary([BII)I (ReadAheadInputStream.java:144)
at com.mysql.jdbc.util.ReadAheadInputStream.read([BII)I (ReadAheadInputStream.java:174)
at com.mysql.jdbc.MysqlIO.readFully(Ljava/io/InputStream;[BII)I (MysqlIO.java:3008)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3469)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(Lcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3459)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(I)Lcom/mysql/jdbc/Buffer; (MysqlIO.java:3900)
at com.mysql.jdbc.MysqlIO.checkErrorPacket()Lcom/mysql/jdbc/Buffer; (MysqlIO.java:873)
at com.mysql.jdbc.MysqlIO.nextRow([Lcom/mysql/jdbc/Field;IZIZZZLcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/ResultSetRow; (MysqlIO.java:1996)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(JIIZ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/RowData; (MysqlIO.java:3410)
at com.mysql.jdbc.MysqlIO.getResultSet(Lcom/mysql/jdbc/StatementImpl;JIIIZLjava/lang/String;Z[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:470)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:3112)
at com.mysql.jdbc.MysqlIO.readAllResults(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:2341)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;Ljava/lang/String;Lcom/mysql/jdbc/Buffer;IIIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods; (MysqlIO.java:2736)
at com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (ConnectionImpl.java:2484)
at com.mysql.jdbc.PreparedStatement.executeInternal(ILcom/mysql/jdbc/Buffer;ZZ[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (PreparedStatement.java:1858)
at com.mysql.jdbc.PreparedStatement.executeQuery()Ljava/sql/ResultSet; (PreparedStatement.java:1966)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery()Ljava/sql/ResultSet; (DelegatingPreparedStatement.java:83)
at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery()Ljava/sql/ResultSet; (SQLProcessor.java:527)
at org.ofbiz.core.entity.GenericDAO.createEntityListIterator(Lorg/ofbiz/core/entity/jdbc/SQLProcessor;Ljava/lang/String;Lorg/ofbiz/core/entity/EntityFindOptions;Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/List;Ljava/util/List;Ljava/util/List;Lorg/ofbiz/core/entity/GenericDAO$TableCleanUp;)Lorg/ofbiz/core/entity/EntityListIterator; (GenericDAO.java:877)
at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(Lorg/ofbiz/core/entity/model/ModelEntity;Lorg/ofbiz/core/entity/EntityCondition;Lorg/ofbiz/core/entity/EntityCondition;Ljava/util/Collection;Ljava/util/List;Lorg/ofbiz/core/entity/EntityFindOptions;)Lorg/ofbiz/core/entity/EntityListIterator; (GenericDAO.java:857)
at org.ofbiz.core.entity.GenericDAO.selectByAnd(Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericDAO.java:729)
at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericHelperDAO.java:166)
at org.ofbiz.core.entity.GenericDelegator.findByAnd(Lorg/ofbiz/core/entity/model/ModelEntity;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericDelegator.java:909)
at org.ofbiz.core.entity.GenericDelegator.findByAnd(Ljava/lang/String;Ljava/util/Map;Ljava/util/List;)Ljava/util/List; (GenericDelegator.java:887)
at org.ofbiz.core.entity.GenericDelegator.getRelated(Ljava/lang/String;Ljava/util/Map;Ljava/util/List;Lorg/ofbiz/core/entity/GenericValue;)Ljava/util/List; (GenericDelegator.java:1505)
at org.ofbiz.core.entity.GenericDelegator.getRelatedByAnd(Ljava/lang/String;Ljava/util/Map;Lorg/ofbiz/core/entity/GenericValue;)Ljava/util/List; (GenericDelegator.java:1452)
at org.ofbiz.core.entity.GenericValue.getRelatedByAnd(Ljava/lang/String;Ljava/util/Map;)Ljava/util/List; (GenericValue.java:317)
at com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssue(Ljava/lang/String;Lorg/ofbiz/core/entity/GenericValue;)Ljava/util/List; (DefaultIssueManager.java:395)
at com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssueObject(Ljava/lang/String;Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (DefaultIssueManager.java:410)
at com.atlassian.jira.issue.managers.RequestCachingIssueManager.getEntitiesByIssueObject(Ljava/lang/String;Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (RequestCachingIssueManager.java:155)
at com.atlassian.jira.issue.comments.CommentSearchManager.getComments(Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (CommentSearchManager.java:106)
at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (DefaultCommentManager.java:174)
at com.atlassian.jira.issue.index.DefaultCommentRetriever.apply(Lcom/atlassian/jira/issue/Issue;)Ljava/util/List; (DefaultCommentRetriever.java:38)
at com.atlassian.jira.issue.index.DefaultCommentRetriever.apply(Ljava/lang/Object;)Ljava/lang/Object; (DefaultCommentRetriever.java:29)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(Lcom/atlassian/jira/issue/Issue;)Ljava/util/Collection; (DefaultIssueIndexer.java:573)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (DefaultIssueIndexer.java:570)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(Lcom/atlassian/jira/issue/Issue;Lcom/atlassian/jira/issue/index/IssueIndexingParams;)Lcom/atlassian/jira/issue/index/DefaultIssueIndexer$Documents; (DefaultIssueIndexer.java:562)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(Lcom/atlassian/jira/issue/Issue;Lcom/atlassian/jira/task/context/Context$Task;)Lcom/atlassian/jira/index/Index$Result; (DefaultIssueIndexer.java:383)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(Ljava/lang/Object;Lcom/atlassian/jira/task/context/Context$Task;)Lcom/atlassian/jira/index/Index$Result; (DefaultIssueIndexer.java:372)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$null$2(Lcom/atlassian/jira/issue/index/DefaultIssueIndexer$IndexOperation;Lcom/atlassian/jira/issue/Issue;Lcom/atlassian/jira/task/context/Context$Task;)Lcom/atlassian/jira/index/Index$Result; (DefaultIssueIndexer.java:311)
at com.atlassian.jira.issue.index.DefaultIssueIndexer$$Lambda$2337.get()Ljava/lang/Object; (Unknown Source)
at com.atlassian.jira.index.SimpleIndexingStrategy.get(Lcom/atlassian/jira/util/Supplier;)Lcom/atlassian/jira/index/Index$Result; (SimpleIndexingStrategy.java:7)
at com.atlassian.jira.index.SimpleIndexingStrategy.get(Ljava/lang/Object;)Ljava/lang/Object; (SimpleIndexingStrategy.java:5)
at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call()Lcom/atlassian/jira/index/Index$Result; (MultiThreadedIndexingStrategy.java:33)
at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call()Ljava/lang/Object; (MultiThreadedIndexingStrategy.java:31)
at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call()Ljava/lang/Object; (BoundedExecutor.java:68)
at java.util.concurrent.FutureTask.run()V (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:624)
at java.lang.Thread.run()V (Thread.java:748)We can see on stack trace called the com.atlassian.jira.issue.comments.DefaultCommentManager.getComments method. This suggests that while building this index record, Jira is obtaining the comments on issues individually from the database and it eventually causes the OutOfMemoryError.
Contextually, JIRA's comments are stored in the jiraaction database table, where the field that stores the comment in MySQL is of type longtext. As clarified in MySQL's documentation for 11.1.3 String Type Overview, a LONGTEXT can store up to a 4GB value.
To summarize, we need to check the database in order to understand the data size there. To diagnose this, the following SQL query can be run:
select concat(p.pkey,'-',ji.issuenum) as IssueID, p.pname as ProjectName, SUM(length(ja.actionbody)) as TotalLengthBytes from project p, jiraissue ji, jiraaction ja where p.ID = ji.PROJECT and ji.ID = ja.issueid group by IssueID order by TotalLengthBytes desc limit 30;
This query might take some time to run, due to its complexity.
The purpose of this SQL query is to show the length of the comments in a Jira instance's database, sorted in descending in order to show the largest comments on top. It'll show just the top 30 in length, and that value can be adjusted if needed.
原因
Huge comments objects are stored on the database and the indexing operation fails to allocate these, leading to the OutOfMemoryError seen. The output of the SQL query above may indicate the size (in bytes) of the objects are stored for single issues. The example below illustrate that the greatest object has a size greater than 3GB.
+------------------+------------------+------------------+
| IssueID | ProjectName | TotalLengthBytes |
+------------------+------------------+------------------+
| EX-1 | Example Project | 3261804965 |
| EX-2 | Example Project | 1440212209 |
| TT-3 | Test Project | 1239449727 |
+------------------+------------------+------------------+回避策
This is recommended to attempt deleting the issues listed on the SQL output that have 1GB or more of total size.
Due to the expensive delete operation, the UI may be unresponsive and a re-index is required on such scenarios, to avoid data inconsistencies.
Another alternative is to use the equivalent REST API call to perform this operation.
Last modified on Mar 6, 2019
Powered by Confluence and Scroll Viewport.