Bamboo remote agents lose connection because of 'No space left on device' error
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Bamboo remote agents lose connection and unable to reconnect.
The following appears in the atlassian-bamboo-agent.log
1
2
3
4
5
6
7
8
9
10
11
12
13
INFO | jvm 1 | 2012/02/07 23:58:48 | 2012-02-07 23:58:48,514 WARN [Scheduler_Worker-7] [JmsTemplateMessageSender] JmsException when trying to send message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@115ea42. Retrying...
INFO | jvm 1 | 2012/02/07 23:58:48 | org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
...
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: java.lang.Throwable: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
...
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.io.RandomAccessFile.writeBytes(Native Method)
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.io.RandomAccessFile.write(RandomAccessFile.java:482)
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:387)
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216)The following appears in the atlassian-bamboo.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
2020-12-04 16:47:51,239 ERROR [ActiveMQ Transport: tcp:///1.1.1.1:49810@8080] [MessageDatabase] KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND
java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
...
2020-12-04 16:47:51,526 ERROR [ActiveMQ Connection Executor: vm://bamboo#0] [BambooServerSimpleMessageListenerContainer] Encountered non-recoverable JMSException
javax.jms.JMSException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1960)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:278)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)
... 6 more
Caused by: org.apache.activemq.broker.BrokerStoppedException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:205)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.aCause
There's no space left on the disk where $BAMBOO_HOME is located. This will cause the Bamboo messaging broker to shutdown with an unrecoverable exception as its unable to journal messages.
Resolution
Bamboo must be restarted to recover the broker, but first the disk space issue should be addressed.
Stop Bamboo
Clean up or add more storage to the Bamboo server where
$BAMBOO_HOMEis located.Start Bamboo
In versions Bamboo 6.10.5 or greater, you may also change the location of the ActiveMQ Journal with the below system property if you want to store it on a different location to where $BAMBOO_HOME is:
1
-Dbamboo.jms.store.directory=/path/to/jms-storeWas this helpful?