Jira reindex fails due to Underlying file changed by an external force
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
During reindexing, warning messages may manifest in one of two ways:
Indexing may fail outright with the following stack error:
2020-03-14 12:46:15,671 IssueIndexer:thread-7 WARN 43588877 729x318x1 admin 10.43.31.107 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '4057058'
2020-03-14 12:46:15,673 IssueIndexer:thread-7 WARN 43588877 729x318x1 admin 10.43.31.107 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] this IndexWriter is closed
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:897)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:911)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1725)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-01-11T07:00:00Z, (lock=NativeFSLock(path=/app/jira/atlassian/application-data/jira/caches/indexesV1/issues/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-14T12:33:40Z))
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.index.ConcurrentMergeScheduler$1.createOutput(ConcurrentMergeScheduler.java:288)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.<init>(Lucene50PostingsWriter.java:105)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:162)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)2. Indexing succeeds, but logs show one or more occurrences of the following error:
2020-04-05 16:42:02,473-0400 NodeReindexServiceThread:thread-1 WARN [c.a.jira.index.AccumulatingResultBuilder] Underlying file changed by an external force at 2020-04-05T00:48:22Z, (lo
ck=NativeFSLock(path=/app/atlassian/application-data/jira/caches/indexesV1/entities/portalpage/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=20
20-04-05T00:48:22.743922Z))3. Indexing fails with a java.nio.file.NoSuchFileException for the write.lock file:
ERROR cio-dm-automation [c.k.r.sil.impl.AbstractSimpleLanguageService] Exception while executing SIL program >>projects/XXXX/addWatchers.sil<<: [SIL Error on line: 4, column: 17] org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
com.keplerrominfo.sil.lang.SILInfoException: [SIL Error on line: 4, column: 17] org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at com.keplerrominfo.sil.lang.SILUtils.error(SILUtils.java:131)
at com.keplerrominfo.sil.lang.SILUtils.throwAway(SILUtils.java:114)
...
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:897)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:911)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
....
Caused by: java.nio.file.NoSuchFileException: /u01/appl/jira/caches/indexesV1/issues/write.lock
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)Environment
Jira Data Center 8.x and above.
Diagnosis
For either of the above scenarios, the relevant error is as follows:
Underlying file changed by an external force at 2020-01-11T07:00:00Z, (lock=NativeFSLock(path=/app/jira/atlassian/application-data/jira/caches/indexesV1/issues/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-14T12:33:40Z))Or:
Caused by: java.nio.file.NoSuchFileException: <JIRA_HOME>/caches/indexesV1/issues/write.lockCause
The exception above indicates that the file write.lock creation time has changed since the re-index started, or the file was removed entirely. This indicates either a problem with the underlying file system or that another process has updated the lock file's creation time or deleted it. The most common causes include:
Antivirus software scans the $JIRA_HOME/caches/indexesV1 (or indexesV2 for Jira 9 and above) directory during a reindex.
External scripts or cron jobs (e.g., cleanup scripts) that delete the indexesV1(or indexesV2 for Jira 9 and above) folder or its contents while Jira is running.
Storing Jira indexes on unsupported network filesystems like NFS or RHGS.
The Jira server is not synchronized to an NTP server.
Solution
If you're utilizing antivirus software, exclude the $JIRA_HOME/caches directory from scanning.
Check for any scheduled scripts or cron jobs that might be modifying or deleting the indexesV1(or indexesV2) directory, and disable them.
Ensure Jira indexes are stored on a server-local file system, as per Jira supported platforms.
Ensure your Jira instance is synced to an NTP server for accurate timekeeping.
To clear the error:
Stop your Jira instance
If still present, delete the write.lock file mentioned in the error
Restart the Jira instance
Kick off a new reindexing operation
Was this helpful?