症状

All notification emails from Jira (batched notifications, non-batched notifications, customer notifications) are piling up in the queue and are sent with a very long delay.

診断

• Looking at atlassian-jira-outgoing-mail debug logs, we can see that each email is being sent with a long delay (20 seconds in the example below):

  grep 'was sent with Message-Id' atlassian-jira-outgoing-mail.log:
  
  2020-07-27 04:53:41,256+0200 DEBUG [] Sending mailitem To='email1@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@24d56930' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32599.1595818401102@Atlassian.JIRA>
  2020-07-27 04:54:01,456+0200 DEBUG [] Sending mailitem To='email2@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@3b62f1d5' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32600.1595818441274@Atlassian.JIRA>
  2020-07-27 04:54:21,639+0200 DEBUG [] Sending mailitem To='email3@test.com' Subject='Updated: (ABC-123)' From='null' FromName='Test User (Jira)' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='null' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@19310596' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service Message was sent with Message-Id <JIRA.428775.1594212067000.32601.1595818481472@Atlassian.JIRA>
  

• The root cause from Jira notifications piling up in the mail queue due to IPv6 issues on JVM has already been checked. Basically, the customer is already using the IPv4 look up method by using the following JVM parameter:

  -Djava.net.preferIPv4Stack=true
  

• Taking thread dumps while the problem is happening will reveal that the mail threads are taking a long time in the method java.net.Inet4AddressImpl.lookupAllHostAddr:

  "Sending mailitem To='email3@test.com' Subject='[ABC-123] - Some Jira ticket' From='null' FromName='Some User' Cc='null' Bcc='null' ReplyTo='email3@test.com' InReplyTo='null' MimeType='text/plain' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@2cb1caa8' MessageId='null' ExcludeSubjectPrefix=true'" #241 daemon prio=5 os_prio=0 tid=0x00007ff58eba6000 nid=0x4361 runnable [0x00007ff54665a000]
     java.lang.Thread.State: RUNNABLE
  	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
  	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
  	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
  	at java.net.InetAddress.getLocalHost(InetAddress.java:1500)
  	- locked <0x00000004d2ea2e30> (a java.lang.Object)
  	at com.sun.mail.smtp.SMTPTransport.getLocalHost(SMTPTransport.java:274)
  	- locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport)
  	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:730)
  	- locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport)
  	at javax.mail.Service.connect(Service.java:388)
  	- locked <0x00000006c7d20388> (a com.sun.mail.smtp.SMTPTransport)
  	at javax.mail.Service.connect(Service.java:246)
  	at javax.mail.Service.connect(Service.java:195)
  	at com.atlassian.mail.server.impl.SMTPMailServerImpl.sendWithMessageId(SMTPMailServerImpl.java:185)
  	at com.atlassian.mail.queue.SingleMailQueueItem.send(SingleMailQueueItem.java:44)
  

• Download this java class onto the Jira Server,  run the following command from the same folder where this class was downloaded and check the time it takes for the Jira server to resolve its own hostname. In the example below, the "real" value is showing "20 seconds", which is extremely slow would delay the sending of each email by at least 20 seconds:

  time <JAVA_BIN_FOLDER>/java GetLocalHostName -Djava.net.preferIPv4Stack=true
  
  Your current IP address : jiraserver/172.10.10.10
  Your current Hostname : jiraserver
  
  real    0m20.148s
  user    0m0.091s
  sys     0m0.063s
  

原因

In order to send notification emails, Jira needs to perform a handshake with the SMTP server, during which it needs to resolve the server hostname. Since the local DNS server is taking longer than it should to resolve Jira server's hostname, this adds a huge delay to the sending of each single Jira notification (since each Jira opens a new connection with the SMTP server to send each single notification).

ソリューション

Engage with the Jira server admin and the network admin to investigate why the DNS resolution takes so much time on the Jira server.

回避策

A possible workaround is to have your Jira server hosts file updated with the hostname and IP address of the server Jira is running on:

• To know which hostname and IP address to add to the hosts file, check the hostname and IP address returned by the command "time <JAVA_BIN_FOLDER>/java GetLocalHostName -Djava.net.preferIPv4Stack=true" executed during the diagnosis steps above. Then add the following line in the hosts file (/etc/hosts for Linux):

  <IP_ADDRESS> <SERVER_HOSTNAME>
  

• If we take the example provided in the diagnosis steps, the line to add to the hosts file will be:

  172.10.10.10 jiraserver