Jira reindex fails due to Underlying file changed by an external force
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
要約
During reindexing, warning messages may manifest in one of two ways:
- Indexing may fail outright with the following stack error:
2020-03-14 12:46:15,671 IssueIndexer:thread-7 WARN 43588877 729x318x1 admin 10.43.31.107 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '4057058'
2020-03-14 12:46:15,673 IssueIndexer:thread-7 WARN 43588877 729x318x1 admin 10.43.31.107 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] this IndexWriter is closed
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:897)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:911)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1725)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-01-11T07:00:00Z, (lock=NativeFSLock(path=/app/jira/atlassian/application-data/jira/caches/indexesV1/issues/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-14T12:33:40Z))
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.index.ConcurrentMergeScheduler$1.createOutput(ConcurrentMergeScheduler.java:288)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.<init>(Lucene50PostingsWriter.java:105)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:162)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)2. Indexing succeeds but logs show one or more occurrences of the following error:
2020-04-05 16:42:02,473-0400 NodeReindexServiceThread:thread-1 WARN [c.a.jira.index.AccumulatingResultBuilder] Underlying file changed by an external force at 2020-04-05T00:48:22Z, (lo
ck=NativeFSLock(path=/app/atlassian/application-data/jira/caches/indexesV1/entities/portalpage/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=20
20-04-05T00:48:22.743922Z))環境
Applicable to Jira 8+
診断
For either of the above scenarios, the relevant error is as follows:
Underlying file changed by an external force at 2020-01-11T07:00:00Z, (lock=NativeFSLock(path=/app/jira/atlassian/application-data/jira/caches/indexesV1/issues/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-14T12:33:40Z))原因
The exception above indicates that the file write.lock creation time has changed since the re-index started, which indicates either a problem with the underlying file system or another process has updated the creation time of the lock file. The most common cause of this is antivirus software scanning the $JIRA_HOME/caches/indexesV1 (or indexesV2 for Jira 9) directory during a reindex.
Additionally, the problem may also intermittently occur if you're storing the Jira indexes in NFS or RHGS, or if your Jira server is not synchronized to an NTP server.
ソリューション
- If you're utilizing antivirus software, exclude the $JIRA_HOME/caches directory from scanning.
- Ensure Jira indexes are stored on a server-local filesystem, as per Jira supported platforms.
- Ensure your Jira instance is synced to an NTP server for accurate timekeeping.
- To clear the error:
- Stop your Jira instance
- If still present, delete the write.lock file mentioned in the error
- Restart the Jira instance
- Kick off a new reindexing operation