要約

Jira Service Management processes incoming mails in two steps: with the Mail Puller Job and the Mail Processor Job. One of the reasons for incoming JSM mails to not be processed is when one of these jobs are failing.

This KB is for troubleshooting a problematic incoming mail (possibly corrupted somehow) that is causing the Mail Processor job to run into a null pointer exception error.

環境

Jira Service Management

診断

The most observable symptom of this issue is when JSM incoming mails are not being processed anymore by mail channels.

Upon checking the incoming mail log atlassian-jira-incoming-mail.log*, there are entries similar to below where the message Error getting sender domain from message 0. Error: java.lang.NullPointerException occurs right before a MailJobRunner failed entry.

2023-05-15 08:36:13,054+1000 WARN [] Caesium-1-4 anonymous     Error getting sender domain from message 0. Error: java.lang.NullPointerException
2023-05-15 08:36:13,057+1000 ERROR [] Caesium-1-4 anonymous     MailJobRunner Failed
java.lang.NullPointerException
	at com.atlassian.jira.internal.mail.processor.feature.processor.AllowBlockListHelper.getSenderEmail(AllowBlockListHelper.java:122) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.internal.mail.processor.feature.processor.AllowBlockListHelper.isSenderBlockListed(AllowBlockListHelper.java:113) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.internal.mail.processor.feature.processor.AllowBlockListHelper.isMessageInBlockList(AllowBlockListHelper.java:101) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.inter
{conal.mail.processor.feature.processor.MailFilterWorker.filter(MailFilterWorker.java:64) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.processMailItem(MailProcessorWorker.java:120) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.lambda$process$0(MailProcessorWorker.java:96) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
...

As a consequence, the Mail Processor job cannot proceed with processing further incoming mails retrieved by the Mail Puller job.

原因

The Error getting sender domain from message 0. Error: java.lang.NullPointerException error occurs while JSM is trying to retrieve the sender's email that is stored in the mail chunk table. This exception happens if the raw encoded email has null sender field.

ソリューション

Identify the Problematic Mail Item

1. Enable debug logging for incoming mails in the JIRA application:
   1. Navigate to Jira Administration ⚙ > System > Logging and Profiling
   2. Click Enable for incoming mail logging and also Enable Debugging, for incoming mail.
      1. Alternate method: If these options are unavailable, click Configure debugging for a package, then turn on DEBUG level for the log package com.atlassian.mail.incoming (for incoming).
   3. Once debugging is turned on, the incoming mail log will be more verbose, which will allow us to identify the mail item causing the null pointer exception.
2. Search the logs for the mail item id causing the error:

• Run the grep command below to retrieve the mail items being processed.

grep -E "Starts processing mail item.*id" $JiraHome/application-logs/atlassian-jira-incoming-mail.log* | grep -Eo "Starts.*" | sort | uniq -c

Ideally, each mail item should only be processed once. If you see more than one occurrence of a mail item being processed, this suggests the Mail Processor job has been trying to process it repeatedly.

In the example below, the Mail Processor Job tried to process the mail item with id=123456 four times.

$ grep -E "Starts processing mail item.*id" $JiraHome/application-logs/atlassian-jira-incoming-mail.log* | grep -Eo "Starts.*" | sort | uniq -c
      4 Starts processing mail item [id: 123456] ...
      1 Starts processing mail item [id: 123457] ...
      1 Starts processing mail item [id: 123458] ...
      1 Starts processing mail item [id: 123459] ...
      1 Starts processing mail item [id: 123460] ...
      1 Starts processing mail item [id: 123461] ...
      1 Starts processing mail item [id: 123462] ...

• Examine the incoming mail log, zooming in on the specific mail item id suspected. We can see in the example below the same Caesium thread attempting to process mail item id 123456 failing with the NullPointerException almost immediately.

In this example, the mail item causing the error is mail item id=123456.

2023-05-15 10:54:14,316+1000 DEBUG [] Caesium-1-4 anonymous     Starts processing mail item [id: 123456] ...
2023-05-15 10:54:14,316+1000 DEBUG [] Caesium-1-4 anonymous     Checking if mail item [id: 123456] is a mail loop ...
2023-05-15 10:54:14,317+1000 DEBUG [] Caesium-1-4 anonymous     Filtering message [message number: 0] using mail filter allow list...
2023-05-15 10:54:14,317+1000 WARN [] Caesium-1-4 anonymous     Error getting sender domain from message 0. Error: java.lang.NullPointerException
2023-05-15 10:54:14,317+1000 DEBUG [] Caesium-1-4 anonymous     Message [message number: 0] is not in allow list
2023-05-15 10:54:14,317+1000 DEBUG [] Caesium-1-4 anonymous     Filtering message [message number: 0] using mail filter block list...
2023-05-15 10:54:14,320+1000 ERROR [] Caesium-1-4 anonymous     MailJobRunner Failed
java.lang.NullPointerException
	at com.atlassian.jira.internal.mail.processor.feature.processor.AllowBlockListHelper.getSenderEmail(AllowBlockListHelper.java:122) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.internal.mail.processor.feature.processor.AllowBlockListHelper.isSenderBlockListed(AllowBlockListHelper.java:113) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]
	at com.atlassian.jira.internal.mail.processor.feature.processor.AllowBlockListHelper.isMessageInBlockList(AllowBlockListHelper.java:101) [jira-email-processor-plugin-5.7.0-REL-0001.jar:?]

Skip/Modify the Offending Mail Item entry

Verify that the mail item is incompletely processed.

1. Run the queries below on your database to check confirm if the mail item was incompletely processed.

   select * from "AO_2C4E5C_MAILITEMAUDIT" where "MAIL_ITEM_ID" = <mailitemID>;
   select * from "AO_2C4E5C_MAILITEMCHUNK" where "MAIL_ITEM_ID" = <mailitemID>;
   select * from "AO_2C4E5C_MAILITEM" where "ID" = <mailitemID>;

2. If the mail item entry was incompletely processed (Mail Processor job crashed midway), the results would look like:

• • AO_2C4E5C_MAILITEMAUDIT result is empty because the processing never completed.
  • AO_2C4E5C_MAILITEMCHUNK would have an email chunk with the missing sender header.
  • AO_2C4E5C_MAILITEM would contain the status of the mailitem and is set to NEW. The FROM_ADDRESS column is null/empty. Its likely that this is where Jira was failing while it was trying to retrieve the from address from the raw mail chunk in the AO_2C4E5C_MAILITEMCHUNK table.

Change the status of the problematic mail item to FAILED. 

1. Verify the status of the mailitem id=123456 before the change

   SELECT * from "AO_2C4E5C_MAILITEM" where "ID"=123456;

2. Update the status of the problematic DB entry to FAILED.

   UPDATE "AO_2C4E5C_MAILITEM" SET "STATUS" = 'FAILED' where "ID" = 123456;

3. Verify the status of the mailitem id=123456 after the change

   SELECT * from "AO_2C4E5C_MAILITEM" where "ID"=123456;

Upon changing the status of the offendint mail item, the Mail Processor job can then skip the errored entry and continue to process all the other remaining mail items.