Remove abandoned or offline nodes in Jira Data Center

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

The nodes that show as active but not alive (that is with no heartbeat) or offline can clutter up your cluster information. In Jira 8.9 we improved the way cluster information is displayed however, you still needed to manually remove stale or offline nodes from the cluster.

That behavior changed in Jira 8.10 and later where stale nodes are automatically removed. (tick)

Pre-Jira 8.9Jira 8.9 以降


Removing an offline node 

If you run Jira 7.13 and later you do not need to stop or restart all running nodes to remove an offline one. It takes a few seconds for the running nodes to respond to changes introduced with REST (If you are not familiar with REST API, here are some examples) or using SQL directly on the database. 

Remove offline nodes using REST (for Jira 8.1 and later)

  1. Make the following request for Jira to return a list of all nodes:

GET /rest/api/2/cluster/nodes

レスポンスの例...
[
 {
  "nodeId": "node3",
  "state": "ACTIVE",
  "lastStateChangeTimestamp": 1551080754927,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": true
 },{
  "nodeId": "node1",
  "state": "OFFLINE",
  "lastStateChangeTimestamp": 1551080840469,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": false
 }
]

2. If you have nodes showing as OFFLINE, delete each offline node from the cluster
DELETE /rest/api/2/cluster/node/{your_offline_node_Id}

3. Double check that the offline node is now deleted and does not show up

GET /rest/api/2/cluster/nodes

For more on these REST API calls, see Jira REST API

Remove offline nodes with database queries

  1.  (optional) Double-check the number of offline nodes by running the command: 
select NODE_ID from clusternode where NODE_STATE ='OFFLINE'; 

2. Copy the node_id  of the offline node and run the following command to delete it:

delete from clusternode where node_id=<id from first query>;


Removing a stale node with no heartbeat

Nodes might show up active but in fact, have no heartbeat if they have been killed abruptly. We advise stopping Jira gracefully to avoid that situation. 

How do I stop Jira gracefully?

Before you remove a VM or AWS instance from the Data Center cluster:

  1. Shut down Jira by using the stop-jira.sh script in <JIRA_INSTALL>/bin or stopping its service.
  2. Then kill the AWS instance or power off the VM. 
If you want to remove a stale node you need to stop Jira on all cluster nodes first. 

You can either remove a stale node using REST API (available for Jira 8.1 and later) or do it manually. 


Remove stale nodes using REST (for Jira 8.1 and later)

  1. Make the following request for Jira to return a list of all nodes:
    GET /rest/api/2/cluster/nodes

    レスポンスの例...
    [
     {
      "nodeId": "node3",
      "state": "ACTIVE",
      "lastStateChangeTimestamp": 1551080754927,
      "ip": "localhost",
      "cacheListenerPort": 40001,
      "nodeBuildNumber": 801000,
      "nodeVersion": "8.1.0-SNAPSHOT",
      "alive": true
     },{
      "nodeId": "node1",
      "state": "ACTIVE",
      "lastStateChangeTimestamp": 1551080840469,
      "ip": "localhost",
      "cacheListenerPort": 40001,
      "nodeBuildNumber": 801000,
      "nodeVersion": "8.1.0-SNAPSHOT",
      "alive": false
     }
    ]
  2. Copy the nodeId's of the nodes that are "ACTIVE" and "alive"=false. 
  3. Ping the nodes to make sure they are not running in your environment and will not back to the cluster in the future.
  4. Make the following request to move the stale node to the offline state
    PUT /rest/api/2/cluster/node/{your_stale_node_Id}/offline
  5. Run GET /rest/api/2/cluster/nodes to make sure the node is offline. 
  6. Delete the stale node from the cluster:
    DELETE /rest/api/2/cluster/node/{your_stale_node_Id}
  7. Double check that the stale node is now deleted and does not show up:
    GET /rest/api/2/cluster/nodes

    For more on these REST API calls, see Jira REST API

Remove old nodes with database queries

データベースの変更を行う場合は 必ず事前にバックアップを取得してください。可能な場合はテスト サーバーで変更を試すことをおすすめします。

  1. Check the database tables and find all rows related to old nodes:

    select * from clusternode;
    select * from clusternodeheartbeat;
  2. Copy the node_id of the stale node and run the command:

    delete from clusternode where node_id = '<node_id>';
    delete from clusternodeheartbeat where node_id = '<node_id>';
  3. Clean old replication records:

    // validate if this is necessary 
    select count(id) from replicatedindexoperation where node_id = '<node_id>';
    // delete
    delete from replicatedindexoperation where node_id = '<node_id>';


    Related tickets

最終更新日 2022 年 9 月 27 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.