ActiveMQ KahaDBStore error when remote agent executes job in Bamboo, causes Bamboo to shutdown
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When a remote agent attempts to execute a job, the Bamboo server crashes. Logs indicate an IOException in the ActiveMQ KahaDB message store, which causes the underlying message broker to fail and take down the Bamboo server.
Environment
Bamboo 10.2
Diagnosis
Start the Bamboo server.
Trigger a job assigned to a remote agent.
Observe the following in the Bamboo server logs just before the crash:
2025-04-14 13:06:33,119 ERROR [ActiveMQ BrokerService[bamboo] Task-4] [KahaDBStore] Failed to load message at: 1:15894
java.io.IOException: Unexpected error on journal read at: 1:15894
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:30) ~[activemq-client-5.18.3.jar:5.18.3]
at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1442) ~[activemq-kahadb-store-5.18.3.jar:5.18.3]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:719) ~[activemq-kahadb-store-5.18.3.jar:5.18.3]
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:787) ~[activemq-kahadb-store-5.18.3.jar:5.18.3]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:707) ~[activemq-kahadb-store-5.18.3.jar:5.18.3]
at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:110) ~[activemq-broker-5.18.3.jar:5.18.3]
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:127) ~[activemq-broker-5.18.3.jar:5.18.3]Cause
The target location for the KahaDB files was corrupted or restricted.
Solution
Redirect the JMS message store to a clean and writable location to avoid corrupted or restricted journal files.
Create a persistent directory for JMS store.
sudo mkdir -p /opt/bamboo/jms-storeSetup correct permissions.
sudo chown bamboo_user:bamboo_group /opt/bamboo/jms-storeUpdate the system property (<INSTALL_DIRECTORY>/bin/setenv.sh).
# Occasionally Atlassian Support may recommend that you set some specific JVM arguments. You can use this variable below to do that. # : ${JVM_SUPPORT_RECOMMENDED_ARGS:="-Dbamboo.jms.store.directory=/opt/bamboo/jms-store"}Restart Bamboo.
Resources
Was this helpful?