Troubleshoot a reindex failure in Jira server
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
問題
After attempting a reindex, the following appears in the atlassian-jira.log and on the JIRA index page:
2013-11-28 20:45:16,765 JiraTaskExectionThread-3 ERROR admin 1243x1446x1 12q9trp 127.0.0.1 /secure/admin/jira/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED. Indexer: DefaultIndexManager: paths: [/home/XX/XX/XXX/XXX/jirahome/caches/indexes/comments, /XX/XX/XXX/XXX/jirahome/caches/indexes/issues, /XX/XX/XXX/XXX/jirahome/caches/indexes/changes]
com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors
at com.atlassian.jira.index.AccumulatingResultBuilder$CompositeResult.await(AccumulatingResultBuilder.java:147)
at com.atlassian.jira.issue.index.DefaultIndexManager.doIndexIssuesInBatchMode(DefaultIndexManager.java:877)
at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:844)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:307)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:272)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
When JIRA applications perform a reindex, errors can happen throughout that process. If an error occurs, exceptions will typically be thrown for individual issues. This means the index can fail for a single issue; however, it does not fail for the entire instance because one issue is causing problems. If the above exception and stack trace is displayed after indexing has finished, it means one or more of the issues failed during the process; however, the entire process did not. It's highly likely that part of the instance has been successfully indexed. Tracking back through the logs prior to the stack trace being thrown will often show exceptions being thrown as issues are indexed.
Some of the known causes for indexing failure are below:
- There is a NULL value under the
issuenum
column ofjiraissue
table - JSWSERVER-16473Getting issue details... STATUS - There is an invalid
issuelinktype
in the database. - Entries in the changegroup table in the database with null values in the created column.
- Entries in the changeitem table in the database with null values in the field column.
- There is a corruption in the workflow post-function that came from third-party plugin.
- If you are using the Charting plugin, you might be running into
- There is a failure while restoring an XML backup or as part of a CSV import which kicks of an index during import. A bug report for this was raised here - JRA-36857Getting issue details... STATUS .
- Sometimes Windows Defender or Anti-Virus softwares throw a false positive quarantining the file created during the indexing process, causing it to fail. The $JIRA_HOME and $JIRA_INSTALL directories should be removed from the antivirus scanning as described in Configure Anti-virus software for use with Jira server KB.
- There are orphaned sub-tasks in the database.
- An unsupported database.
診断
Identifying the specific issue that is identified in the error can be determined by searching the following keyword in the atlassian-jira.log
which contains full traces of the re-indexing process:
Indexing failed for Issue
java.lang.NullPointerException exception thrown as a warning message with no other error message in the indexing process.
IssueIndexer:thread-6 WARN mr80639 466x686x1 1jdr0h0 10.56.14.1,10.237.54.143 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
at com.atlassian.jira.index.DefaultIndex$Failure.<init>(DefaultIndex.java:100)
......
Caused by: java.lang.NullPointerException
com.atlassian.rm.teams.customfields.team.IssueTeamService.getTeamForNonSubTask(IssueTeamService.java:56)
原因 1
- NULL value under the
issuenum
column ofjiraissue
table - Delete an issue while it is being processed by a bulk edit, creates a null issuenum entry in jiraissue table - JSWSERVER-16473Getting issue details... STATUS
Run the following SQL query against JIRA's database to check if there are any issues with a NULL in the issuenum column:
SELECT id,issuenum,project FROM jiraissue WHERE issuenum IS NULL;
原因 2
invalid
issuelinktype
in the database
Run the following SQL query against JIRA's database to check if there are any invalid issuelink type associations in the database:
SELECT * FROM issuelink WHERE linktype NOT IN (SELECT id FROM issuelinktype);
原因 3
- the changegroup table contains NULL values in the created column
Run the following SQL query against JIRA's database to check for NULL entries in the created column of the changegroup table:
SELECT * FROM changegroup WHERE created IS NULL ORDER BY issueid;
原因 4
- the changeitem table contains NULL values in the field column
Run the following SQL query against JIRA's database to check for NULL entries in the field column of the changeitem table:
All these entries with changeitem.field=null are related to custom fields. However, since the field is empty, it is not possible to determine if they are related to a plugin or not.
SELECT ji.id, ji.issuenum, ji.project, cg.*, ci.*
FROM jiraissue ji
INNER JOIN changegroup cg ON ji.id = cg.issueid
LEFT JOIN changeitem ci ON cg.id = ci.groupid
where ci.field IS NULL;
There is also a use case where there will be NULL ID values in the changegroup table rather than NULL values in the field column in the changeitem table if the above query returns records. You can verify if this is the case if the following query returns no records for the field column of the changeitem table:
select ci.* from changeitem as ci where ci.field is null;
原因 5
- A corrupted workflow due to a post-function provided by third-party Add-on
Check for third-party Add-ons used in workflows and possible incompatible versions installed. See Resolution for more details.
原因 6
- There is a bug in the JIRA Charting Plugin, tracked as part of
原因 7
原因 9
原因 10
- An unsupported database; for example MySQL Maria DB or latest PostgreSQL versions that are not yet supported by the JIRA version you're running. Please always check the Jira supported platforms article for your version of JIRA (you can choose the version in the top right corner) before upgrading the database
Getting more data
If none of the solutions below fit your scenario, then you need to enable extra logging. For that, change the Logging Level of the package com.atlassian.jira.index.AccumulatingResultBuilder
from WARN to DEBUG on Logging & Profiling (Administration > System > Logging & Profiling). This will enable JIRA to write more information about the errors found on the logs. If you are not sure how to analyse or interpret the extra logging, get in touch with Atlassian Support and attach a Support Zip so our Support Engineers can help you.
First thing to try:
Restart the JIRA application.
- Do a full Re-index, by choosing the option to 'Lock and Re-Index'.
- Go into Safe Mode and retest
Specifically for Data center environment, ensure that the Data Center nodes are linked to an NTP server and that time is identical on those nodes.
If this does not help, please contact Atlassian Support
ソリューション
ソリューション 1
If the query from the diagnosis returns any results, run the following to remove the NULL entries:
DELETE FROM jiraissue WHERE issuenum IS NULL;
ソリューション 2
If the query from the diagnosis returns any results, run the following to remove any invalid entries:
DELETE FROM issuelink WHERE linktype NOT IN (SELECT id FROM issuelinktype);
ソリューション 3
If the query from the diagnosis returns any results, run the following to fix it by inserting a date where null occurs:
UPDATE changegroup SET CREATED='2015-01-01 00:00:00' WHERE CREATED IS NULL;
ソリューション 4
If the query from the diagnosis returns any results, run the following to fix it by deleting the entries where column field=null :
DELETE FROM changeitem WHERE id IN ( SELECT ci.id FROM changeitem ci INNER JOIN changegroup cg on cg.id = ci.groupid WHERE ci.field IS NULL );
Since the column field is empty in database table changeitem, the reference to determine the history of the change was lost. We can only see an entry of change, it's not possible to determine the field it is related to.
If the query above fails with the error ERROR 1093 (HY000): You can't specify target table 'changeitem' for update in FROM clause in MySQL Database, you can run the query below:
DELETE FROM changeitem WHERE id IN ( select * from ( SELECT ci.id FROM changeitem ci INNER JOIN changegroup cg on cg.id = ci.groupid WHERE ci.field IS NULL ) as t);
If the second query from the diagnosis of Cause 4 returns no result but the first query does, run the following SQL query on DBs other than MySQL to delete the entries from the changegroup table rather than the changeitem table where there is a NULL ID instead:
DELETE FROM changegroup WHERE id IN ( SELECT cg.id FROM jiraissue ji INNER JOIN changegroup cg ON ji.id = cg.issueid LEFT JOIN changeitem ci ON cg.id = ci.groupid where ci.id IS NULL );
For MySQL DB, we'll need to wrap the select in another select. Run the below query instead:
DELETE FROM changegroup WHERE id IN ( select * from ( SELECT cg.id FROM jiraissue ji INNER JOIN changegroup cg ON ji.id = cg.issueid LEFT JOIN changeitem ci ON cg.id = ci.groupid where ci.id IS NULL) as t );
ソリューション 5
- If you have confirmed the use of a third-party plugin - such as, ScriptRunner - being used as part of a workflow post-function
- Delete the post-function
- Uninstall the plugin.
- Re-index JIRA
- Re-install the plugin
- Add back the post-function after the indexing is completed.
- See Related KBs as well for more details:
ソリューション 6
- Upgrade to a later version of the JIRA Charting Plugin that includes a fix for
ソリューション 7
- Upgrade to a later version of JIRA that includes a fix for - JRA-36857Getting issue details... STATUS
ソリューション 8
- Create an exception in Windows Defender for the indexes directory and all its content.
ソリューション 9
- Follow the steps documented in the workaround session of bug - JPOSERVER-2015Getting issue details... STATUS
ソリューション 10
- If the unsupported database is a result of the DB server upgrade, you may need to revert the DB upgrade back to the supported version: Supported platforms