ReIndexing reporting wrong numbers on Jira Datacenter with Archived projects
プラットフォームについて: Data Center のみ - この記事は、Data Center プラットフォームのアトラシアン製品にのみ適用されます。
この KB は Data Center バージョンの製品用に作成されています。Data Center 固有ではない機能の Data Center KB は、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
問題
Jira Datacenter is reporting wrong numbers after finishing foreground re-Indexing.
The re-Indexing finishes unexpectedly before it reaches 100% and no errors are being logged.
The following appears in the atlassian-jira.log:
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing started
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultNodeReindexService] Pausing node re-index service
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] ReindexAll in foreground: {indexIssues=true, indexChangeHistory=true, indexComments=true, indexWorklogs=true, forceReloadFromDatabase=false}
.....
.....
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 68% complete. Current index:
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 3840094ms. Reindex run: 101
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finishedAlso the below could be seen in the logs:
2019-01-1 12:35:00,166 http-nio-8080-exec-1 DEBUG user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] isIndexConsistent: Issue: expectedCount=235000; actualCount=159000
2019-01-1 12:35:00,166 http-nio-8080-exec-1 WARN user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=235000; actualCount=159000As can be seen the Index is having a big discrepancy from the number of issues got from the database (expectedCount).
診断
環境
Jira Datacenter 7.10.x or later
Having 1 or more projects in Archive.
Diagnostic Steps
Enable Debug logs on the below packages:
com.atlassian.jira.index.AccumulatingResultBuilder com.atlassian.jira.issue.index
This should allow Jira to list all the individual issues being indexed and we could verify how many issues are being indexed exactly:
$ egrep "111x11111x1.*Indexing issue:" atlassian-jira.log* |wc -l
159000
$This would then needs to be compared with total number of issues in the Jira database minus the amount of issues in the Archived projects.
Another diagnostic step is to check the output of the below rest API call:
GET <JIRA_BASE_URL>/rest/api/2/index/summary
This call will report the number of issues in Index, in the database and in Archive, Jira stores those number in a special cache. Those number are then used to drive the reporting of the re-Indexing operation.
This cache is being repopulated if a project is marked for Archive or if it is brought back from Archive.
The healthy output should look like this:
{"nodeId":"jira_node1","reportTime":"2019-01-1T15:50:00.356+0100","issueIndex":
{"indexReadable":true,"countInDatabase":235000,"countInIndex":159000,"countInArchive":76000,"lastUpdatedInDatabase":"2019-01-1T00:40:00.357+0100","lastUpdatedInIndex":"2019-01-1T12:35:00.000+0100"}原因
For some currently unknown reason, the Jira issue counts cache might get corrupted and reports a wrong count for the issue count in Archive, this results in wrong reporting for the re-Indexing job and causes confusion.
The summary rest API call will look like this:
{"indexReadable":true,"countInDatabase":235000,"countInIndex":159000,"countInArchive":0,"lastUpdatedInDatabase":"2019-01-1T00:40:00.357+0100","lastUpdatedInIndex":"2019-01-1T12:35:00.000+0100"}The count of issues in Archive can be seen as 0, even though Jira does have issues in the archived projects.
This issue is reported in the following bug report: JRASERVER-69296 - Getting issue details... STATUS
回避策
The workaround for this issue is to correct the values of the cache by flushing it. This should rebuild the number and correct the indexing calculations. This can be done by adding or removing a project from Archive.
Please follow the below steps:
Create new empty project or recover an existing project from Archive.
The size of project doesn't matter since Deindexing issues for archived project is done by a single query to index (shouldn't take more than 100ms)
After restoring a project you don't need to reindex this project.Archive this project again (either archive or restoring of any project from Archive is clearing the issue counts cache)
Verify if countInArchive has changed in /rest/api/2/index/summary response.
注意
If you use the AWS QuickStart templates, scaling a new node may show the same warning after a new node is provisioned. If this is the case, you may need to repeat this for the new nodes by deleting the dummy project we created as part of the workaround and re-creating/archiving. Alternatively, you can create another new project or unarchive/archive an existing project.