Jira notifications piling up in the mail queue due to IPv6 issues on JVM
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Symptoms
Jira is not able to connect to a specific mail server or mail queue keeps getting overloaded as each message takes a long time to leave the queue. As a result all notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue and are sent with a very long delay.
Diagnosis
Looking at atlassian-jira-outgoing-mail debug logs, we can see the message is always taking longer between these two messages below:
1
2
2014-05-06 15:36:06,848 DEBUG [jira.local@domain.com] Sending mailitem To='user@domain.com' Subject='(XXX-1) Summary' From='null' FromName='JIRA updates' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='null' MessageId='null' ServiceRunner Mail Queue Service DEBUG SMTP: connected to host "mail.domain.net", port: 25
2014-05-06 15:36:18,860 DEBUG [jira.local@domain.com] Sending mailitem To='user@domain.com' Subject='(XXX-1) Summary' From='null' FromName='JIRA updates' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='null' MessageId='null' ServiceRunner Mail Queue Service EHLO jira.domain.net
The following error can also appear on the outgoing mail log files:
1
2
3
4
5
2015-02-04 21:00:26,605 WARN [Mail Server - JIRA] QuartzScheduler_Worker-0 ServiceRunner TEST TEST[10100]:
javax.mail.MessagingException: Connect failed while connecting to host 'mailserver.example.com' as user 'user@example.com' via protocol 'smtp': javax.mail.MessagingException: Connect failed;
nested exception is:
java.net.UnknownHostException: mailserver.example.com
javax.mail.MessagingException: Connect failed;
Taking Generating a Thread Dump while the problem is happening will reveal the mail threads are taking a long time in the method java.net.Inet6AddressImpl.lookupAllHostAddr:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
"Sending mailitem To='email3@test.com' Subject='[ABC-123] - Some Jira ticket' From='null' FromName='Some Name' Cc='null' Bcc='null' ReplyTo='email3@test.com' InReplyTo='null' MimeType='text/plain' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@2cb1caa8' MessageId='null' ExcludeSubjectPrefix=true'" #241 daemon prio=5 os_prio=0 tid=0x00007ff58eba6000 nid=0x4361 runnable [0x00007ff54665a000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
at java.net.InetAddress.getLocalHost(Unknown Source)
- locked <0x00000007802e1cc8> (a java.lang.Object)
at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:235)
- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:646)
- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
at javax.mail.Service.connect(Service.java:295)
- locked <0x00000007d61a5e58> (a com.sun.mail.smtp.SMTPTransport)
(...) Cause
The IPv6 lookup method can enter an infinite loop when a certain race condition occurs. This occurs infrequently, however can cause deadlocks where some threads end up blocked.
Please check the Java bug: https://bugs.openjdk.java.net/browse/JDK-8158038 for more information.
Solution
Resolution
Setting Properties and Options on Startup on JIRA startup:
1-Djava.net.preferIPv4Stack=trueWith the parameter stated above set to true, Java will not execute IPv6 lookup methods and the problem should be avoided.
Please ensure you have your JIRA server hosts file updated with JIRA server hostname and IP addresses, both IPv4 and IPv6.
Was this helpful?