Bamboo Remote Agents fail to start with "Software caused connection abort: recv failed" message
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
When installing or running a Bamboo Remote agent on JDK version 11 or 8u341 and later, it refuses to connect to the Bamboo's ActiveMQ JMS broker service port with a "Software caused connection abort: recv failed" (Java 11) or without an exception and instead with a generic "Connection or outbound has closed, continuing to retry." (Java 8) or message when over SSL.
Java 11, 17 and 21:
Agent logs
1
2
3
INFO | jvm 8 | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,601 INFO [AgentRunnerThread] [AgentRegistrationBean] Registering agent on the server,
INFO | jvm 8 | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,602 ERROR [ActiveMQ Task-1] [FailoverTransport] Failed to connect to [ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=300000] after: 10 attempt(s)
INFO | jvm 8 | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,617 INFO [ActiveMQ Connection Executor: unconnected] [PooledConnectionFactory] Expiring connection ActiveMQConnection {id=ID:EC2AMAZ-JQCAG7H-49860-1658475932964-1:1,clientId=ID:EC2AMAZ-JHKSJDH-49860-1658475932964-0:1,started=false} on IOException: Software caused connection abort: recv failedJava 8u341+:
Agent logs
1
2
INFO | jvm 8 | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,601 INFO [AgentRunnerThread] [AgentRegistrationBean] Registering agent on the server,
INFO | jvm 8 | 2022/06/22 15:47:08 | 2022-06-22 05:47:08,617 WARN [ActiveMQ Task-1] [FailoverTransport] Failed to connect to [ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=300000] after: 1 attempt(s) with Connection or outbound has closed, continuing to retry.Server logs
1
2
2022-06-22 15:47:08,577 WARN [ActiveMQ BrokerService[bamboo] Task-2049] [TransportConnector] Could not accept connection from tcp://X.X.X.X:49871: No available authentication scheme (No available authentication scheme)
2022-06-22 15:47:22,365 WARN [ActiveMQ BrokerService[bamboo] Task-2049] [TransportConnector] Could not accept connection from tcp://X.X.X.X:49874: Remote host terminated the handshake (SSL peer shut down incorrectly)Environment
Bamboo Server 8+ with secure agents (SSL) and the
bamboo.manage.jms.sslproperty is set totrue(default)Bamboo Remote Agent running over an updated JDK 21, JDK 17, JDK 11 or JDK 8u341 or later
The Bamboo server comes from older releases such as 5 or 6 up to 8 and contains an old
<bamboo-home>/shared/configuration/broker.kskeystore file with an old certificate
Diagnosis
The agent tries to connect to the Bamboo Server ActiveMQ JMS port and fails the SSL negotiation.
After enabling the following debug properties on the Bamboo Agent application, along with Java's -Djavax.net.debug=ssl,handshake on the Agent's wrapper.conf:
For Bamboo version 8 and earlier
conf/log4j.properties
1
2
3
4
5
6
7
8
log4j.rootLogger=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d %p [%t] [%c{1}] %m%n
log4j.category.com.atlassian.bamboo=DEBUG
log4j.category.com.atlassian.util.profiling.UtilTimerStack=ALL
log4j.category.org.apache.sshd=DEBUG
log4j.category.org.apache.activemq=DEBUGconf/wrapper.conf
1
2
wrapper.java.additional.4=-Dlog4j.configuration=<bamboo-agent-home>/conf/log4j.properties
wrapper.java.additional.5=-Djavax.net.debug=ssl,handshakeFor Bamboo version 9 and later
conf/log4j2.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
packages = com.atlassian.bamboo.log
status = warn
rootLogger=INFO, filelog, console
appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{DEFAULT} %p [%t] [%C{1}] %m%n
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = OFF
appender.filelog.type = RollingFile
appender.filelog.name = filelog
appender.filelog.fileName = ../atlassian-bamboo-agent.log
appender.filelog.filePattern = atlassian-bamboo.log-agent.%i
appender.filelog.layout.type = PatternLayout
appender.filelog.layout.pattern = %d{DEFAULT} %p [%t] [%C{1}] %m%n
appender.filelog.policies.type = Policies
appender.filelog.policies.size.type = SizeBasedTriggeringPolicy
appender.filelog.policies.size.size = 100MB
appender.filelog.strategy.type = DefaultRolloverStrategy
appender.filelog.strategy.max = 5
appender.filelog.strategy.fileIndex = min
logger.bamboo-added.level=DEBUG
logger.bamboo-added.name=com.atlassian.bamboo
logger.bamboo-profiling.level=ALL
logger.bamboo-profiling.name=com.atlassian.util.profiling.UtilTimerStack
logger.bamboo-sshd.level=DEBUG
logger.bamboo-sshd.name=org.apache.sshd
logger.bamboo-activemq.level=DEBUG
logger.bamboo-activemq.name=org.apache.activemqconf/wrapper.conf
1
2
wrapper.java.additional.4=-Dlog4j2.configurationFile=<bamboo-agent-home>/conf/log4j2.properties
wrapper.java.additional.5=-Djavax.net.debug=ssl,handshakeWe can see the following SSL handshake issue:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
INFO | jvm 10 | 2022/06/22 17:06:58 | 2022-06-22 17:06:58,549 INFO [AgentRunnerThread] [AgentRegistrationBean] Current agent remote definition: agent7 Remote agent on host agent7
INFO | jvm 10 | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,632 INFO [AgentRunnerThread] [AgentRegistrationBean] Registering agent on the server,
INFO | jvm 10 | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,801 DEBUG [ActiveMQ Task-1] [FailoverTransport] urlList connectionList:[ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000], from: [ssl://bamboo.example.net:46804?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000]
INFO | jvm 10 | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,898 DEBUG [ActiveMQ Task-1] [FailoverTransport] Attempting 0th connect to: ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000
INFO | jvm 10 | 2022/06/22 17:06:59 | javax.net.ssl|DEBUG|4C|ActiveMQ Task-1|2022-06-22 17:06:59.908 AEST|Utilities.java:73|the previous server name in SNI (type=host_name (0), value=bamboo.example.net) was replaced with (type=host_name (0), value=bamboo.example.net)
INFO | jvm 10 | 2022/06/22 17:06:59 | 2022-06-22 17:06:59,913 DEBUG [ActiveMQ Task-1] [WireFormatNegotiator] Sending: WireFormatInfo { version=12, properties={TcpNoDelayEnabled=true, SizePrefixDisabled=false, CacheSize=1024, ProviderName=ActiveMQ, StackTraceEnabled=true, PlatformDetails=Java, CacheEnabled=true, TightEncodingEnabled=true, MaxFrameSize=9223372036854775807, MaxInactivityDuration=90000, MaxInactivityDurationInitalDelay=10000, ProviderVersion=5.16.3}, magic=[A,c,t,i,v,e,M,Q]}
(...)
INFO | jvm 10 | 2022/06/22 17:07:00 | javax.net.ssl|ERROR|4C|ActiveMQ Task-1|2022-06-22 17:07:00.090 AEST|TransportContext.java:345|Fatal (HANDSHAKE_FAILURE): Couldn't kickstart handshaking (
INFO | jvm 10 | 2022/06/22 17:07:00 | "throwable" : {
INFO | jvm 10 | 2022/06/22 17:07:00 | javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1696)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:921)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1291)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:123)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1019)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:149)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
INFO | jvm 10 | 2022/06/22 17:07:00 | at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/java.lang.Thread.run(Thread.java:829)
INFO | jvm 10 | 2022/06/22 17:07:00 | Caused by: java.io.EOFException: SSL peer shut down incorrectly
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
INFO | jvm 10 | 2022/06/22 17:07:00 | at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
INFO | jvm 10 | 2022/06/22 17:07:00 | ... 19 more}
INFO | jvm 10 | 2022/06/22 17:07:00 |
INFO | jvm 10 | 2022/06/22 17:07:00 | )
INFO | jvm 10 | 2022/06/22 17:07:00 | 2022-06-22 17:07:00,097 DEBUG [ActiveMQ Task-1] [FailoverTransport] Connect fail to: ssl://bamboo.example.net:54663?socket.verifyHostName=false&wireFormat.maxInactivityDuration=90000, reason: {}We can also observe that Bamboo's ActiveMQ Keystore contains a certificate created long ago:
Bamboo's ActiveMQ broker.ks
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# keytool -list -keystore shared/configuration/broker.ks -storepass bamboo -v
Keystore type: jks
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: jmsbrokerkey
Creation date: 24/03/2016
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=Bamboo ActiveMQ Broker
Issuer: CN=Bamboo ActiveMQ Broker
Serial number: 1ff5b90bc6c5aa
Valid from: Thu Mar 24 15:14:33 AEDT 2016 until: Wed Jun 22 14:14:33 AEST 2016
Certificate fingerprints:
SHA1: 43:AD:5D:CA:44:61:DD:79:GG:8A:14:DA:A0:C2:78:94:1C:64:4D:4F
SHA256: 7C:5B:C7:67:61:93:29:11:AB:85:B6:99:67:2E:C8:23:06:BA:40:B4:07:94:46:C5:3A:10:FD:62:DA:DB:6A:1B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 1024-bit DSA key (weak)
Version: 3
*******************************************
*******************************************
Warning:
<jmsbrokerkey> uses a 1024-bit DSA key which is considered a security risk. This key size will be disabled in a future update.
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore shared/configuration/broker.ks.very-old.certificate -destkeystore shared/configuration/broker.ks.very-old.certificate -deststoretype pkcs12".Cause
JDK 21, JDK17, JDK 11 and JDK 8u341 or later use TLSv1.3 whenever possible. On certain occasions, when the Bamboo Server's ActiveMQ broker.ks keystore and its certificate were created using older Java versions and DSA algorithms, JDK versions 21, 17, 11 and 8u341 or later will fail to negotiate a connection over TLSv1.3, even though it is advertised as compatible.
Solution
This is a mandatory step if the current ActiveMQ Broker certificate is of type DSA. To find out if the Bamboo Server uses a DSA certificate please run the following command and look for Public Key Algorithm: dsaEncryption:
openssl s_client -connect <bamboo_server>:<jms_port> <<< "Q" | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | openssl x509 -text | grep 'Public Key Algorithm:'
Once confirmed, there are two possible solutions to this:
Option 1: Regenerate the Bamboo Server's ActiveMQ certificate
This solution (Option 1) requires you to delete the existing trust store files on every remote agent that has already connected to your Bamboo Server so that it can trust the new certificate which can be a significant amount of manual work depending on how many agents you have and how they are provisioned.
For extra detail, please see How to generate a new Bamboo ActiveMQ Broker Certificate
Summary of the steps:
Completely stop Bamboo
Backup and delete the
<bamboo-home>/shared/configuration/broker.ksfileStart the Bamboo Server; a new
broker.ksfile will be createdOn each Remote Agent, delete the
<bamboo-agent-home>/configuration/jmsclient.*files so they can request the new certificate and properties from the Server upon reconnectionMake sure that the Java version running on the Remote Agents align with the version running on the Bamboo Server
Restart the Remote Agents
Option 2: Configure Bamboo's ActiveMQ JMS broker service to listen to TLSv1.2 only
Completely stop Bamboo
Apply thetransport.enabledProtocols changes as suggested on How to restrict cryptographic protocols used by Bamboo's JMS broker for remote agent communication (item #3) to instruct the Agents to use TLSv1.2
Start the Bamboo Server
Restart the Remote Agents in case they don't reconnect automatically
Was this helpful?