Indexer stops with ERROR java.io.IOException: No space left on device
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Newly added content is unsearchable because Indexer stopped working after system ran out of memory.
Bellow are some of the exceptions that can be found in atlassian-confluence.log
1
2
3
4
5
6
ERROR [Indexer: 1] [internal.index.lucene.LuceneBatchIndexer] doIndex Exception indexing document with id: 12345678
│java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:474)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)1
2
3
4
5
6
ERROR [scheduler_Worker-10] [org.quartz.core.ErrorLogger] schedulerError Job (DEFAULT.IndexQueueFlusher threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: com.atlassian.bonnie.LuceneException: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries]
...
Caused by: com.atlassian.confluence.api.service.exceptions.ServiceException: Failed to process entries
...
Caused by: java.io.IOException: No space left on device1
2
3
4
5
6
7
8
WARN [scheduler_Worker-10] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable
java.io.IOException: No space left on device
WARN [scheduler_Worker-1] [search.lucene.tasks.AddDocumentIndexTask] perform Error getting document from searchable
java.io.FileNotFoundException: /opt/confluence/confluence-home/index/_z8ey.fdx (No space left on device)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:466)Diagnosis
On Linux
Check free disk space and inode count for <confluence-home>/index directory
1
2
# df -h
# df -iIncrease disk capacity or analyze which directories / files are large and can be removed
If you ran out of inodes, you might need to migrate to another file system or partition, see 'Not enough space on disk' errors in Confluence 6.5 or later when disk space is available
On Windows
Check free disk space from properties of the drive
Check free disk space from PowerShell
1
Get-WmiObject -Class Win32_logicaldiskIncrease space on disk partition or free up space by removing unnecessary and large files
Cause
As documented in 'java.io.IOException No space left on device' due to disk space KB, "No space left on device" exception indicates that there is not enough space left for Confluence to write inside <confluence-home> directory.
Besides failures to generate backups, or save attachments, indexing tasks will also fail.
Solution
Confluence stores it's index locally, inside home directory, and it needs to have enough free space allocated to continue indexing content.
After enough space is freed, rebuild index from scratch by following one of the bellow procedures:
How to Rebuild the Content Indexes From Scratch on Confluence Data Center
How to Rebuild the Content Indexes From Scratch on Confluence Server
Lucene index must be stored locally on the server and can't be stored on NFS mount as outlined in Health Check: Lucene index files location
Was this helpful?