Unable to search issues and the log shows read past EOF messages in Jira
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
要約
Several operations in Jira may fail or take too much time to perform, like loading dashboards, gadgets, filters creating or updating issues.
In the atlassian-jira.log we see a read past EOF error message.
環境
All versions of Jira Data Center.
診断
Jira's behavior becomes erratic and it may take a long time to load some screens, but the easiest way to diagnose this scenario is:
- Accessing an issue's direct URL loads the issue successfully:
JIRA-BASE-URL/browse/JIRA-123, yet trying to perform a search doesn't bring any result — if the screens loads at all. - On the
atlassian-jira.logwe can see several "read past EOF" messages. Some of the scenarios include the ones below.
During Jira startup
2021-07-12 14:25:25,064 JIRA-Bootstrap ERROR [c.a.jira.index.LuceneCorruptionChecker] Can't open index '/opr/atlassian/jira/caches/indexes/issues'. Reason:
java.io.IOException: read past EOFAccessing certain JIRA pages
SEVERE: Internal server error
com.atlassian.jira.issue.index.SearchUnavailableException: com.atlassian.util.concurrent.LazyReference$InitializationException: com.atlassian.jira.util.RuntimeIOException: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _ow2: fieldsReader shows 1 but segmentInfo shows 50
at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:85)
at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:76)
at com.atlassian.jira.issue.index.SearcherCache.retrieveIssueSearcher(SearcherCache.java:38)Searching for issues or accessing a Jira filter
Caused by: java.io.FileNotFoundException: /local/jira/jira444/home/caches/indexes/entities/searchrequest/_o5.fnm (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)Trying to reindex Jira
java.io.IOException: read past EOF
Task completed in 2 seconds with unexpected error.
Started Today 8:29 PM.
Finished Today 8:29 PM.
com.atlassian.jira.util.RuntimeIOException: java.io.IOException: read past EOF at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:145) at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:44) at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:170) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:219) ... at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:249) at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: read past EOFまたは
2012-11-25 16:21:36,897 JiraTaskExectionThread-1 ERROR admin 981x203x1 6slf04 10.1.250.116 /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED. Indexer: DefaultIndexManager: paths: [/var/atlassian/application-data/jira/caches/indexes/comments, /var/atlassian/application-data/jira/caches/indexes/issues, /var/atlassian/application-data/jira/caches/indexes/changes]
com.atlassian.jira.util.RuntimeIOException: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null
at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:146)
at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:43)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:205)
at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:824)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:335)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:301)
Caused by: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null原因
This EOF error is a consequence of corrupted Lucene index files. Jira's unable to recover on it's own from this corruption and any operations fail therefor.
This is likely to happen if Jira's process was killed abruptly while writing to the index files or if the machine or VM itself has been interrupted. Storage failures may also corrupt the index files if they were being written at the time.
ソリューション
The solution is to restore the Indexes or rebuild them from scratch.
If you're running a Jira Data Center with two or more nodes, you may copy the indexes from a healthy node to the corrupted one through the Admin interface, instead.
Restoring indexes
If you have a backup from the index folder (JIRA-HOME/caches/indexes), you may follow this procedure:
- Jira を停止します。
- Move all the contents from
JIRA-HOME/caches/indexesto a backup folder just for the sake of it - Restore the contents of a
indexesbackup folder intoJIRA-HOME/caches/indexes - Jira の起動
- Confirm it's operational and dashboards, gadgets and filters are working fine — albeit without recent data (only the data up to when the index backup was taken)
- Start a background reindex so the more recent data is gradually updated in the index
Rebuilding index from scratch
If there's no recent indexes folder backup, the only solution is to lock and reindex Jira. Unfortunately this will cause a downtime for Jira users until the reindex is complete.
- Jira を停止します。
Make a backup of
JIRA_HOME/caches/indexesand it's sub-directoriesRemove all the contents from
JIRA_HOME/caches/indexes- Jira を起動します。
- Perform a locked/full reindex