Unable to search issues and the log shows read past EOF messages in Jira
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Several operations in Jira may fail or take too much time to perform, like loading dashboards, gadgets, filters creating or updating issues.
In the atlassian-jira.log we see a read past EOF error message.
Environment
All versions of Jira Data Center.
Diagnosis
Jira's behavior becomes erratic and it may take a long time to load some screens, but the easiest way to diagnose this scenario is:
Accessing an issue's direct URL loads the issue successfully:
JIRA-BASE-URL/browse/JIRA-123, yet trying to perform a search doesn't bring any result — if the screens loads at all.On the
atlassian-jira.logwe can see several "read past EOF" messages. Some of the scenarios include the ones below.
During Jira startup
1
2
2021-07-12 14:25:25,064 JIRA-Bootstrap ERROR [c.a.jira.index.LuceneCorruptionChecker] Can't open index '/opr/atlassian/jira/caches/indexes/issues'. Reason:
java.io.IOException: read past EOFAccessing certain JIRApages
1
2
3
4
5
SEVERE: Internal server error
com.atlassian.jira.issue.index.SearchUnavailableException: com.atlassian.util.concurrent.LazyReference$InitializationException: com.atlassian.jira.util.RuntimeIOException: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _ow2: fieldsReader shows 1 but segmentInfo shows 50
at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:85)
at com.atlassian.jira.issue.index.DefaultIndexManager$1.get(DefaultIndexManager.java:76)
at com.atlassian.jira.issue.index.SearcherCache.retrieveIssueSearcher(SearcherCache.java:38)Searching for issues or accessing a Jira filter
1
2
3
4
5
Caused by: java.io.FileNotFoundException: /local/jira/jira444/home/caches/indexes/entities/searchrequest/_o5.fnm (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:78)
at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:108)Trying to reindex Jira
1
2
3
4
5
6
java.io.IOException: read past EOF
Task completed in 2 seconds with unexpected error.
Started Today 8:29 PM.
Finished Today 8:29 PM.
com.atlassian.jira.util.RuntimeIOException: java.io.IOException: read past EOF at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:145) at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:44) at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:170) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:219) ... at java.util.concurrent.FutureTask.run(FutureTask.java:138) at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:249) at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: read past EOFor
1
2
3
4
5
6
7
8
9
2012-11-25 16:21:36,897 JiraTaskExectionThread-1 ERROR admin 981x203x1 6slf04 10.1.250.116 /secure/admin/IndexReIndex.jspa [jira.util.index.CompositeIndexLifecycleManager] Reindex All FAILED. Indexer: DefaultIndexManager: paths: [/var/atlassian/application-data/jira/caches/indexes/comments, /var/atlassian/application-data/jira/caches/indexes/issues, /var/atlassian/application-data/jira/caches/indexes/changes]
com.atlassian.jira.util.RuntimeIOException: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned null
at com.atlassian.jira.index.DefaultIndexEngine.clean(DefaultIndexEngine.java:146)
at com.atlassian.jira.index.DefaultManager.deleteIndexDirectory(DefaultManager.java:43)
at com.atlassian.jira.issue.index.DefaultIssueIndexer.deleteIndexes(DefaultIssueIndexer.java:205)
at com.atlassian.jira.issue.index.DefaultIndexManager.doStopTheWorldReindex(DefaultIndexManager.java:824)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:335)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:301)
Caused by: java.io.IOException: directory '/var/atlassian/application-data/jira/caches/indexes/comments' exists and is a directory, but cannot be listed: list() returned nullCause
This EOF error is a consequence of corrupted Lucene index files. Jira's unable to recover on it's own from this corruption and any operations fail therefor.
This is likely to happen if Jira's process was killed abruptly while writing to the index files or if the machine or VM itself has been interrupted. Storage failures may also corrupt the index files if they were being written at the time.
Solution
The solution is to restore the Indexes or rebuild them from scratch.
If you're running a Jira Data Center with two or more nodes, you may copy the indexes from a healthy node to the corrupted one through the Admin interface, instead.
Restoring indexes
If you have a backup from the index folder (JIRA-HOME/caches/indexes), you may follow this procedure:
Stop Jira
Move all the contents from
JIRA-HOME/caches/indexesto a backup folder just for the sake of itRestore the contents of a
indexesbackup folder intoJIRA-HOME/caches/indexesStart Jira
Confirm it's operational and dashboards, gadgets and filters are working fine — albeit without recent data (only the data up to when the index backup was taken)
Start a background reindex so the more recent data is gradually updated in the index
Rebuilding index from scratch
If there's no recent indexes folder backup, the only solution is to lock and reindex Jira. Unfortunately this will cause a downtime for Jira users until the reindex is complete.
Stop JIRA
Make a backup of
JIRA_HOME/caches/indexesand it's sub-directoriesRemove all the contents from
JIRA_HOME/caches/indexesStart JIRA
Perform a locked/full reindex
Was this helpful?