Bamboo unable to connect to mail server due to IPv6 issues on JVM
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Symptoms
Bamboo is not able to connect to a mail server or mail queue keeps getting overloaded as each message takes a long time to leave the queue. Test emails work as intended.
Diagnosis
Enable Mail Debug logging in Bamboo
Generate Thread dumps for Bamboo
Looking at Bamboo debug logs, we can see the message "451 Timeout waiting for data from client":
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
DEBUG SMTP: trying to connect to host "email.server.com", port 587, isSSL false
....
DEBUG SMTP: connected to host "email.server.com", port: 587
EHLO email.server.com
451 Timeout waiting for data from client.
HELO email.server.com
DEBUG SMTP: EOF: [EOF]
2019-05-27 15:41:40,214 WARN [mail-queue-processor] [EmailService] Uncaught exception while processing mail queue
com.atlassian.mail.MailException: javax.mail.MessagingException: [EOF]
at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:225)
at com.atlassian.mail.server.impl.SMTPMailServerImpl.send(SMTPMailServerImpl.java:162)
at com.atlassian.bamboo.mail.EmailService.sendEmailSynchronously(EmailService.java:115)
at com.atlassian.bamboo.mail.EmailService$MailQueueProcessor.run(EmailService.java:67)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.mail.MessagingException: [EOF]
at com.sun.mail.smtp.SMTPTransport.issueCommand(SMTPTransport.java:2168)
at com.sun.mail.smtp.SMTPTransport.helo(SMTPTransport.java:1552)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:703)
at javax.mail.Service.connect(Service.java:386)
at javax.mail.Service.connect(Service.java:245)
at javax.mail.Service.connect(Service.java:194)
at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:185)
... 4 more4. The thread dumps will reveal the mail threads are taking a long time in the method java.net.Inet6AddressImpl.lookupAllHostAddr:
1
2
3
4
5
6
7
8
9
10
11
"mail-queue-processor" #49 daemon prio=5 os_prio=0 tid=0x00007ff727563000 nid=0x7a63 runnable [0x00007ff7232f3000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getLocalHost(InetAddress.java:1501)
- locked <0x00000000c0dfb8a0> (a java.lang.Object)
at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:253)
- locked <0x00000000e00ae788> (a com.sun.mail.smtp.SMTPTransport)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:701)
- locked <0x00000000e00ae788> (a com.sun.mail.smtp.SMTPTransport)Cause
The IPv6 lookup method can enter an infinite loop when a certain race condition occurs. This is due to DNS servers or the local network not being configured for IPv6. It can occur infrequently and may cause deadlocks where some threads end up blocked.
Solution
Resolution
Set the following parameter on Bamboo startup:
1-Djava.net.preferIPv4Stack=trueWith the parameter stated above set to true, Java will not execute IPv6 lookup methods and the problem should be avoided.
Was this helpful?