Index snapshot failed due to insufficient space on the storage
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Copying index from node to node is failing with message No space left on device. This is also applicatable for creation of index recovery snapshot.
The following appears in the log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
2016-06-10 08:47:00,311 ClusterMessageHandlerServiceThread:thread-1 ERROR [atlassian.jira.cluster.OfBizMessageHandlerService] There was a problem handling a cluster message
java.lang.RuntimeException: java.io.IOException: No space left on device
at com.atlassian.jira.index.ha.IndexUtils.copySpecificIndex(IndexUtils.java:256)
at com.atlassian.jira.index.ha.IndexUtils.copyIndexes(IndexUtils.java:173)
at com.atlassian.jira.index.ha.IndexUtils.takeIndexSnapshot(IndexUtils.java:127)
at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.copyIndex(DefaultIndexCopyService.java:164)
at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.backupIndex(DefaultIndexCopyService.java:142)
at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.backupIndex(DefaultIndexCopyService.java:124)
...
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
at org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:457)
at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:118)
at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:127)
at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:251)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:107)
at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:3080)
at com.atlassian.jira.index.ha.IndexUtils.copySpecificIndex(IndexUtils.java:247)
... 17 moreDiagnosis
Environment
Different partitions for <JIRA_INSTALL> and <JIRA_HOME> are used.
Cause
During creating the snapshot of the index it is created in java.io.tmpdir directory, by default it set to CATALINA_BASE/temp (<JIRA_INSTALL>/temp).
Solution
Workaround
Set in bin/setenv.sh CATALINA_TMPDIR variable and point to directory at <JIRA_HOME> partition.
You can use a Symbolic Link in a *nix environment.
Resolution
Please check that partition for <JIRA_INSTALL> has enough space.
Was this helpful?