Jira notifications piling up in the mail queue due to a server hostname resolution issue

Summary

Symptoms

All notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue and are sent with a very long delay.

Diagnosis

• Looking at atlassian-jira-outgoing-mail debug logs, we can see that each email is being sent with a long delay (20 seconds in the example below):

  1 2 3 4 5 grep 'was sent with Message-Id' atlassian-jira-outgoing-mail.log: 2020-07-27 04:53:41,256+0200 DEBUG [] Sending mailitem To='email1@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@24d56930' MessageId='null' ExcludeSubjectPrefix=false' anonymous Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32599.1595818401102@Atlassian.JIRA> 2020-07-27 04:54:01,456+0200 DEBUG [] Sending mailitem To='email2@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@3b62f1d5' MessageId='null' ExcludeSubjectPrefix=false' anonymous Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32600.1595818441274@Atlassian.JIRA> 2020-07-27 04:54:21,639+0200 DEBUG [] Sending mailitem To='email3@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@19310596' MessageId='null' ExcludeSubjectPrefix=false' anonymous Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32601.1595818481472@Atlassian.JIRA>

• The root cause from Jira notifications piling up in the mail queue due to IPv6 issues on JVM has already been checked. Basically, the customer is already using the IPv4 look up method by using the following JVM parameter:

  1 -Djava.net.preferIPv4Stack=true

• Taking thread dumps while the problem is happening will reveal that the mail threads are taking a long time in the method java.net.Inet4AddressImpl.lookupAllHostAddr:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 "Sending mailitem To='email3@test.com' Subject='[ABC-123] - Some Jira ticket' From='null' FromName='Some User' Cc='null' Bcc='null' ReplyTo='email3@test.com' InReplyTo='null' MimeType='text/plain' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@2cb1caa8' MessageId='null' ExcludeSubjectPrefix=true'" #241 daemon prio=5 os_prio=0 tid=0x00007ff58eba6000 nid=0x4361 runnable [0x00007ff54665a000] java.lang.Thread.State: RUNNABLE at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getLocalHost(InetAddress.java:1500) - locked <0x00000004d2ea2e30> (a java.lang.Object) at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:274) - locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:730) - locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport) at javax.mail.Service.connect(Service.java:388) - locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport) at javax.mail.Service.connect(Service.java:246) at javax.mail.Service.connect(Service.java:195) at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:185) at com.atlassian.mail.queue.SingleMailQueueItem.send(SingleMailQueueItem.java:44)

Cause

In order to send notification emails, Jira needs to perform a handshake with the SMTP server, during which it needs to resolve the server hostname. Since the local DNS server is taking longer than it should to resolve Jira server's hostname, this adds a huge delay to the sending of each single Jira notification (since each Jira opens a new connection with the SMTP server to send each single notification).

Solution

Engage with the Jira server admin and the network admin to investigate why the DNS resolution takes so much time on the Jira server.

Workaround

A possible workaround is to have your Jira server hosts file updated with the hostname and IP address of the server Jira is running on:

• Add the following line in the hosts file (/etc/hosts for Linux):

  1 <IP_ADDRESS> <SERVER_HOSTNAME>