要約

Infrastructure metrics for JMX monitoring and in-product diagnostic are used in Jira Software Data Center and Jira Service Management Data Center. These metrics help you monitor the health and performance of your instance infrastructure:

• outgoing mail server connection
• incoming mail server connection
• external user directories connectivity
• shared home write latency
• local home write latency
• internode latency

Learn more about other Jira-specific metrics for in-product diagnostics

ソリューション

Outgoing mail server connection state

The connection state metric for the outgoing mail server (SMTP) is created once it’s configured and removed when it’s deleted.

mail.outgoing.connection.state

mail.outgoing.connection.state.custom attempts to connect to a mail server and pings it with the NOOP or RSET SMTP commands. This operation is performed once in a minute. The metric reports failed state when a connection can’t be established or a response for commands is invalid. During the measurement process, the mail server will timeout after 10 seconds and report the disconnected value.

Available custom metric values: connected (true or false), totalFailures (the sum of the false values since the restart).

Incoming mail server connection state

The connection state metric for each incoming mail server is created once it’s added to Jira. Metrics are compatible with all types of mail servers and all authentication types.

mail.incoming.connection.state

mail.incoming.connection.state.custom attempts to connect to a remote server and perform a read-only open the INBOX folder operation. This operation is performed once in a minute.

Only one measuring process can run at a time to avoid overwriting results. Since the mail servers are remote and can have an individually defined connection timeout, during the measurement process, the timeouts are overridden to the standard maximum of 10 seconds.

For servers using basic authentication, the attempt to open a connection will fail if there is a problem. For servers using OAuth, a bad connection is only visible when trying to open the Inbox folder.

When an incoming mail server is deleted, its metric is unregistered too. Incoming mail servers are differentiated by using their names stored as metric tags. When changing the names of mail servers, note that this will create new metrics.

A sample MBean ObjectName for this metric will look as follows: com.atlassian.jira:type=metrics,category00=mail,category01=incoming,category02=connection,category03=state,name=custom,tag.serverName=<mailName>.

The tag tag.serverName will contain the name of your configured incoming mail server, however, every space character will change to _. For example, the mail name Google mail box will change to Google_mail_box.

Available custom metric values: connected (true or false), totalFailures.

External user directories connectivity

Two factors for external user directories are measured:

• Connection state

  • values – true or false

  • totalFailures – the total number of failures 

• レイテンシ

  • value – the current value in milliseconds and statistics

These values are measured for every type of user directory except for the internal user directory. This operation is performed once in a minute. As a key to differentiating user directories metrics, their names stored as metric tags are used. If you change the name of a user directory, a new metric will be created. Swapping names between user directories will adversely affect the readability of metrics.

Disabling or removing user directory will immediately remove its metrics from JMX.

user.directory.connection.latency

user.directory.connection.latency is measured by performing user search on an uncached user directory. The query for users doesn’t specify any parameters or restrictions and the maximum result number is specified to one.

Check the chart with the user.directory.connection.latency.value metric dropping to the -1 value.

Available metrics: value, statistics.

user.directory.connection.state.custom

user.directory.connection.state.custom performs the same check as the Test connection configuration under Administration > System > User directories. The metric is checked differently depending on the type of the user directory:

• LDAP – checks the connection with the machine (connectivity only).

• Internal with LDAP Authentication – checks connection with the machine (connectivity only).

• Active directory – searches users (connectivity and authentication).

• Crowd – searches users (connectivity and authentication).

Check the chart with a 2-hour outage on the external user directory.

Available custom metric values: connected (true or false), totalFailures (the sum of false values).

Shared home write latency

For clustered Data Center instances, the time to write sample data to the shared home is measured. High times of shared home latency will impact attachments, avatars, index snapshots, and other items.

home.shared.write.latency

• home.shared.write.latency measures the time in milliseconds to write a sample file on the shared home. A few measurements are taken every minute to improve metric accuracy.
• home.shared.write.latency.value contains a calculated median latency from the last iteration.
• home.shared.write.latency.statistics contains aggregated statistics from every individual measurement and should give better insight into the outliers and latency distribution.

Only one measuring process can run at a time. Whenever a timeout of 15 seconds per three file writes is breached, the .value metric will be updated with the -1 value. In this case, the shared home can be considered unreachable.

Check the chat with the healthy home.shared.write.latency.statistics metric.

Available metrics: value, statistics

Local home write latency

Local home write latency metrics measure the local disk write performance: home.local.write.latency.synthetic and home.local.write.latency.indexwriter.

High local disk latency will have a significant impact on the Jira performance in cache replication and index persistence.

home.local.write.latency.synthetic

• home.local.write.latency.synthetic measures the time of a synthetic file write operation with a guarantee of persistence on the local home. A few measurements are taken every minute to improve metric accuracy. This metric is designed to closely imitate the persisting of cache replication operations.
• home.local.write.latency.synthetic.value contains a calculated median latency from the last iteration. This value is reported in microseconds for better precision.
• home.local.write.latency.synthetic.statistics contains aggregated statistics from every individual measurement and should give better insight into the outliers and the latency distribution.

If it takes more than five seconds to make seven writes, the local disk is considered unreachable. This should never happen, as the Jira instance would become unusable.

Check the chart with the healthy home.local.write.latency.synthetic.value metric with a latency lower than 2 ms.

Available metrics: value, statistics.

home.local.write.latency.indexwriter

• home.local.write.latency.indexwriter reports the time of flushing the Lucene index buffer to the local disk. The metric is based on the real traffic and represents the current status and performance of the Lucene subsystem. This metric is updated only when the Lucene buffer is persisted, usually after index updates.
• home.local.write.latency.indexwriter.statistics contains aggregated statistics from the measurements.

These metric values don’t reflect pure disk performance. The reported time is highly related to the volume of updated documents and may sporadically report high latency times unrelated to the disk performance.

Check the chart with the healthy home.local.write.latency.indexwriter.statistics metric.

Available metrics: statistics.

ノード間レイテンシー

For clustered Data Center instances, the latency of communication between nodes is measured with the metrics node.latency.statistics and node.connection.state.custom.  The metrics are based on the real traffic and represents the current status and performance of the Cache replication subsystem. High internode latency will impact cache replication and DBR index replication.

The node.latency.statistics metric measures the time in milliseconds to send a cache invalidation message to other nodes through RMI. This can be interpreted as a basic internode communication latency. This metric will be unregistered from JMX when the connection status is set to disconnected, and the metric will appear again when the latency can be measured.

Check the chart with internode latency during the Jira redeployment to new machines.

The node.connection.state.custom metric reports the current connection status to other nodes in the cluster. If the cache was successfully replicated, the status will be reported as connected. If the cache replication mechanism failed due to a network connection issue, the status will be reported as disconnected.

Check the chart with the internode node connection state during the Jira redeployment.

Both metrics are created for every other node in the cluster. The metrics include the tag tag.destNode=<nodeId>.

If a node is shut down properly, you can expect both metrics to disappear from the JMX registry. If the node is unexpectedly unreachable and Jira tries to replicate a cache operation to it, you can expect node.connection.state.custom to report as disconnected.