要約

The cache is not replicating between Jira Nodes.

環境

Jira Data Center using at least 2 nodes.

Any Jira version that supports Data Center.

診断

• The following failure can be found in the Jira Health Check file included in the support zip:

  Name: Cluster Cache Replication
  Is healthy: false
  Failure reason: The node nodeX is not replicating
  Severity: CRITICAL

• The following error can be found in the Jira application logs. Note the space character located between the host IP address "XX.XXX.X.X" and the character ";":

  2020-01-31 07:32:44,006-0400 localq-reader-7 ERROR      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-INVALIDATION] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.propertyset.CachingOfBizPropertyEntryStore.cache', action=REMOVE, key=CacheKey[entityName=ApplicationUser,entityId=13871], value == null ? true, replicatePutsViaCopy=false, creationTimeInMillis=1622456749450} from cache replication queue: [queueId=queue_62b7f2e4a4c84b03a74d49c29a831172_7_4585b5a075e0343372457f1f073d6e92, queuePath=/var/atlassian/application-data/jira/localq/queue_62b7f2e4a4c84b03a74d49c29a831172_7_4585b5a075e0343372457f1f073d6e92], failuresCount: 10/10. Removing from queue. Error: java.rmi.UnknownHostException: Unknown host: XX.XXX.X.X ; nested exception is: 
      	java.net.UnknownHostException: XX.XXX.X.X 
  com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.UnknownHostException: Unknown host: XX.XXX.X.X ; nested exception is: 
  	java.net.UnknownHostException: XX.XXX.X.X 
  	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:90)
  	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:96)
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)
  Caused by: java.rmi.UnknownHostException: Unknown host: XX.XXX.X.X ; nested exception is: 

• When checking the cluster.properties file of the problematic node (located in the local node home folder), a trailing space can be found after the node hostname:

  # This ID must be unique across the cluster
  jira.node.id = nodeX
  # The location of the shared home directory for all Jira nodes
  jira.shared.home = /mounted_directory/jira_shared_home
  # IP Address used by this node for cache replication
  ehcache.listener.hostName = XX.XXX.X.X 
  # Ports used by this node for cache replication
  ehcache.listener.port = 40001
  ehcache.object.port = 40011

• In some cases, the configuration in the cluster.properties file is correct, but the HOSTNAME environment variable is set to a wrong name or IP address

原因

The Jira nodes in the Data Center cluster are unable to reach the node that fails to replicate, because its hostname is invalid in the cluster.properties file (it has a trailing space).

ソリューション

1. Shutdown all the Data Center nodes
2. Edit the ehcache.listener.hostName setting from the cluster.properties file of the problematic node by removing the empty space after the IP address:
   • This file is located at the local <JIRA_HOME> folder of this node
3. Make sure that the HOSTNAME environment variable is set to the appropriate hostname.

4. Update the entry in the clusternode table of the Jira Database to ensure that the empty space is also removed there:

   update clusternode set ip = '<node_IP_address>' where node_id = '<node_ID>';

5. Restart all the Jira nodes