Jira reindex fails due to Underlying file changed by an external force
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
During reindexing, warning messages may manifest in one of two ways:
Indexing may fail outright with the following stack error:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
2020-03-14 12:46:15,671 IssueIndexer:thread-7 WARN 43588877 729x318x1 admin 10.43.31.107 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] Indexing failed for Issue - '4057058'
2020-03-14 12:46:15,673 IssueIndexer:thread-7 WARN 43588877 729x318x1 admin 10.43.31.107 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] this IndexWriter is closed
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:897)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:911)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1725)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1464)
Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2020-01-11T07:00:00Z, (lock=NativeFSLock(path=/app/jira/atlassian/application-data/jira/caches/indexesV1/issues/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-14T12:33:40Z))
at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
at org.apache.lucene.index.ConcurrentMergeScheduler$1.createOutput(ConcurrentMergeScheduler.java:288)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43)
at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.<init>(Lucene50PostingsWriter.java:105)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:162)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:230)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:115)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4443)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)2. Indexing succeeds but logs show one or more occurrences of the following error:
1
2
3
2020-04-05 16:42:02,473-0400 NodeReindexServiceThread:thread-1 WARN [c.a.jira.index.AccumulatingResultBuilder] Underlying file changed by an external force at 2020-04-05T00:48:22Z, (lo
ck=NativeFSLock(path=/app/atlassian/application-data/jira/caches/indexesV1/entities/portalpage/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=20
20-04-05T00:48:22.743922Z))Environment
Applicable to Jira 8+
Diagnosis
For either of the above scenarios, the relevant error is as follows:
1
Underlying file changed by an external force at 2020-01-11T07:00:00Z, (lock=NativeFSLock(path=/app/jira/atlassian/application-data/jira/caches/indexesV1/issues/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2020-03-14T12:33:40Z))Cause
The exception above indicates that the file write.lock creation time has changed since the re-index started, which indicates either a problem with the underlying file system or another process has updated the creation time of the lock file. The most common cause of this is antivirus software scanning the $JIRA_HOME/caches/indexesV1 (or indexesV2 for Jira 9) directory during a reindex.
Additionally, the problem may also intermittently occur if you're storing the Jira indexes in NFS or RHGS, or if your Jira server is not synchronized to an NTP server.
Solution
If you're utilizing antivirus software, exclude the $JIRA_HOME/caches directory from scanning.
Ensure Jira indexes are stored on a server-local filesystem, as per Jira supported platforms.
Ensure your Jira instance is synced to an NTP server for accurate timekeeping.
To clear the error:
Stop your Jira instance
If still present, delete the write.lock file mentioned in the error
Restart the Jira instance
Kick off a new reindexing operation
Was this helpful?