Mail notifications slow or stopped due to IPv6 issues on JVM
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Symptoms
Bitbucket server's mail queue keeps getting overloaded or is not able to connect to a specific mail server. This causes each message to take a long time to leave the queue. Test emails will still work.
Diagnosis
Looking at atlassian-bitbucket-mail.log logs, we can see the mail queue contantly full:
1
ERROR [Caesium-1-4] bitbucket.mail-log Email could not be queued: mail queue is full;Taking thread dumps while the problem is happening will reveal the mail threads are taking a long time in the method java.net.Inet6AddressImpl.lookupAllHostAddr:
1
2
3
4
5
6
7
8
9
10
11
12
13
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
at java.net.InetAddress.getLocalHost(Unknown Source)
- locked <0x00000007802e1cc8> (a java.lang.Object)
at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:235)
- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:646)
- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
at javax.mail.Service.connect(Service.java:295)
- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
(...) Cause
The IPv6 lookup method can enter an infinite loop when a certain race condition occurs. This occurs infrequently, however can cause deadlocks where some threads end up blocked.
Please check the Java bug: https://bugs.openjdk.java.net/browse/JDK-8158038 for more information.
Solution
Resolution
Add the following parameter int the "<bitbucket-install>/bin/_start-webapp.sh" (.bat in windows) file under JVM_SUPPORT_RECOMMENDED_ARGS:
1-Djava.net.preferIPv4Stack=trueWith the parameter stated above set to true, Java will not execute IPv6 lookup methods and the problem should be avoided.
Restart Bitbucket
Was this helpful?