Bamboo or agents fail with "No space left on device" when the partition runs out of inodes
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bamboo or agents report "no space left on device" despite having enough free space on the disk or partition. The following errors can be found in the <bamboo-home>/logs/atlassian-bamboo.log or <bamboo-agent-home>/atlassian-bamboo-agent.log files:
atlassian-bamboo.log / atlassian-bamboo-agent.log
2021-03-24 18:35:49,747 ERROR [http-nio-8085-exec-11577] [FiveOhOh] 500 Exception was thrown.
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.force0(Native Method)
at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76)
at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:388)atlassian-bamboo.log / atlassian-bamboo-agent.log
2021-04-15 17:36:42,034 ERROR [http-nio-8085-exec-138] [FiveOhOh] 500 Exception was thrown.
java.nio.file.FileSystemException: /path/to/file: No space left on device
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)Environment
Any supported version of Bamboo hosted on Linux.
An agent hosted on Linux.
Diagnosis
The following actions can be taken to diagnose the issue and confirm that you're running into the problem described in this article:
Confirm there's enough disk space available in the partition where the <bamboo-home>, <bamboo-shared-home>, or <bamboo-agent-home> folders are located by running the df -h command:
$ df -h Filesystem Size Used Avail Use% Mounted on udev 16G 0 16G 0% /dev tmpfs 3.2G 2.5M 3.2G 1% /run /dev/nvme0n1p2 468G 214G 231G 49% / tmpfs 16G 475M 16G 3% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 16G 0 16G 0% /sys/fs/cgroupConfirm there's enough disk space available by creating a Bamboo support zip and looking for the <free-disk-space> property inside the application.xml file:
<free-disk-space>291 GB</free-disk-space>Look for the inode utilization on the partition where the <bamboo-home>, <bamboo-shared-home>, or <bamboo-agent-home> folders are located by running the df -i command:
$df -i Filesystem Inodes IUsed IFree IUse% Mounted on udev 4070819 700 4070119 1% /dev tmpfs 4078019 1342 4076677 1% /run /dev/nvme0n1p2 31227904 30915624 312280 99% / tmpfs 4078019 559 4077460 1% /dev/shm tmpfs 4078019 5 4078014 1% /run/lock tmpfs 4078019 18 4078001 1% /sys/fs/cgroup /dev/loop3 10817 10817 0 100% /snap/core18/1988 /dev/loop2 10790 10790 0 100% /snap/core18/1997 /dev/loop4 19362 19362 0 100% /snap/dbeaver-ce/110 /dev/loop5 27807 27807 0 100% /snap/gnome-3-28-1804/145 /dev/loop6 11713 11713 0 100% /snap/core20/904 /dev/loop7 3156 3156 0 100% /snap/remmina/4802 /dev/nvme0n1p1 0 0 0 - /boot/efiℹ️ A high percentage in the column "IUse%" (in the partition where Bamboo was performing I/O activity) will indicate inode exhaustion. In this example, the problematic partition is /dev/nvme0n1p2.
Cause
You're running out of inodes on the file system/partition where the <bamboo-home>, <bamboo-shared-home>, or <bamboo-agent-home> folders are located. There are too many files in the partition so when Bamboo or the agent tries to create new ones, even though there is enough disk space, there's no data structure to point to them. A good and detailed explanation of what inodes are can be found in this external link:
Solution
The easiest way to solve this problem is to delete or move files from the affected partition. Start by inspecting where the highest number of inodes comes from in the partition. You can use the following command to count the number of files per folder:
$ cd /path/to/folder/
$ sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -nOnce you identify the folder(s) with the highest contribution to the inode count, you should use one of the following methods to free up inodes, depending on the file type:
Files that are not related to the Bamboo or agent installation: Solution 1
Bamboo's build and deployment results, artifacts, and logs (for example, <bamboo-shared-home>/artifacts): Solution 2
The files located inside the build working directories of your agents at <bamboo-agent-home>/xml-data/build-dir: Solution 3
Solution 1 – Manually deleting or moving files
You can free up some inodes by deleting or moving files that are not related to Bamboo from this partition. Please be cautious when doing so, and make sure to contact the owner of the files before making any changes to ensure they are aware of your actions and to avoid any potential disruptions. Just because the files are not used by Bamboo, it doesn't mean they're not in use by other apps, scripts, etc.
Solution 2 – Adjusting expiry configuration
If build and deployment results, artifacts, or logs contribute to a high inode count, avoid manually removing these files to ensure consistency between Bamboo's database and the file system. Instead, adjust the expiry configuration to automatically remove the files either on a global level or plan level. The folders containing artifacts on the file system do not clearly correspond to the plan name from which the artifacts were generated. This is because they are named with storage tags, not plan keys (for example, plan-18022401). Each plan or plan branch is assigned a unique storage tag. Here's an example of the results from running the find command above inside the <bamboo-shared-home>/artifacts folder:
7442810 ./plan-1230832371
3351969 ./plan-1358482224
3033469 ./plan-1358482239
2995840 ./plan-1358478002
2247089 ./plan-1390283919
1979621 ./plan-1358471273
1821892 ./plan-575222308
1765573 ./plan-1390283929
1440827 ./plan-1358463942
1440635 ./plan-1358467151To identify the Bamboo plan or plan branch responsible for a specific folder, you can run the following SQL query inside the database used by Bamboo:
SELECT FULL_KEY,STORAGE_TAG FROM BUILD WHERE BUILD_TYPE IN ('CHAIN','CHAIN_BRANCH') AND STORAGE_TAG IN ('storageTag1','storageTag2');ℹ️ Replace storageTag with the actual storage tags from the find command results (for example, plan-1230832371).
The results of the query above will look similar to this:
full_key | storage_tag |
|---|---|
SAMPLE-PLAN | plan-1230832371 |
PROJ-PLAN | plan-1358482224 |
After identifying the plans or plan branches with the highest inode counts, consider implementing a stricter expiry configuration. This can be applied globally or within the plan configuration to retain fewer files, thereby reducing the number of inodes used in the file system. This will not have an immediate impact because Bamboo will only delete the files during the next expiry run.
You can check when expiry will run again (or even manually run expiry) from the Bamboo administration > Overview > Plans > Expiry page.
Solution 3 – Cleaning working directory after each build (agent only)
This solution only applies to the following scenarios:
Agents running out of inodes due to a high number of files inside the <bamboo-agent-home>/xml-data folder.
Bamboo Server or Data Center running out of inodes when using local agents due to a high number of files inside the <bamboo-home>/local-working-dirfolder.
This solution does NOT apply to the following scenario:
Bamboo Server or Data Center running out of inodes, but only remote, elastic, or ephemeral agents are in use, with no local agents.
If the largest number of files is coming from the <bamboo-agent-home>/xml-data folder (you can check this using the find command mentioned above), you can choose to enable the following option inside your plans. This will not have an immediate impact as Bamboo will only clear the working directory after new builds have run. However, this should help reduce and maintain the size of the data folder to an acceptable level, thereby reducing the number of inodes used in the file system.
Enable the Clean working directory after each build option from the job configuration to ensure that no files are left on the build working directory after the job is complete.
Was this helpful?