Confluence triggers cluster panic while updating the database - Unable to acquire JDBC Connection
プラットフォームについて: Data Center のみ - この記事は、Data Center プラットフォームのアトラシアン製品にのみ適用されます。
この KB は Data Center バージョンの製品用に作成されています。Data Center 固有ではない機能の Data Center KB は、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
要約
An error message appears when trying to access Confluence:
Database is being updated by an instance which is not part of the current cluster.環境
Confluence Data Center
診断
When accessing Confluence, some nodes will throw errors in the logs, making the application inaccessible. When looking at the atlassian-confluence.log, will find entries like below:
2021-05-25 01:50:42,150 WARN [http-nio-8090-exec-9 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921699643, threadId=403, threadName='http-nio-8090-exec-9 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 3990689cfb713b1a | userName: anonymous
2021-05-25 01:50:42,151 WARN [http-nio-8090-exec-59 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921701223, threadId=13882, threadName='http-nio-8090-exec-59 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 98643738cbe68254 | userName: anonymous
2021-05-25 01:50:42,154 WARN [http-nio-8090-exec-13 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921736283, threadId=411, threadName='http-nio-8090-exec-13 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: e9b11240652a7c46 | userName: anonymous
2021-05-25 01:50:42,154 WARN [http-nio-8090-exec-69 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921781252, threadId=25435, threadName='http-nio-8090-exec-69 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: e2396d2c8415c734 | userName: anonymous
2021-05-25 01:50:42,156 WARN [http-nio-8090-exec-46 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921721253, threadId=623, threadName='http-nio-8090-exec-46 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 0eba7f5425983dab | userName: anonymous
2021-05-25 01:50:42,158 WARN [http-nio-8090-exec-68 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921779637, threadId=25434, threadName='http-nio-8090-exec-68 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 20f56684c4c282c3 | userName: anonymous
2021-05-25 01:50:42,162 WARN [http-nio-8090-exec-11 url: /status] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621921733614, threadId=408, threadName='http-nio-8090-exec-11 url: /status', userId='<unknown>', type='web-request', summary='/status'}
-- url: /status | traceId: 37fdb3f28d2429ed | userName: anonymous
2021-05-25 01:51:01,158 WARN [hz.confluence.cached.thread-30] [internal.cluster.impl.ClusterHeartbeatManager] log [142.136.96.235]:5801 [CONFLUENCE_APP_NCW] [3.11.6] Suspecting Member [142.136.96.236]:5801 - 1a46fbe9-da3a-4469-be93-e2250c7a73f5 because it has not sent any heartbeats since 2021-05-25 01:50:00.516. Now: 2021-05-25 01:51:01.158, heartbeat timeout: 60000 ms, suspicion level: 1.00
Following that, the error below will be spammed in the application logs, until the connection with the database is re-established.
2021-05-25 04:25:23,567 WARN [Caesium-1-4] [impl.schedule.caesium.JobRunnerWrapper] runJob Scheduled job EhCacheCompactionJob#EhCacheCompactionJob completed unsuccessfully with response JobRunnerResponse[runOutcome=FAILED,message='CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
GenericJDBCException: Unable to acquire JDBC Connection
SQLException']原因
This is caused when a network glitch or maintenance happens, breaking the communication between the confluence server and the database.
When in a clustered instance, after the nodes re-establish the communication with the database, only the first node will be recognized as part of the cluster, and all of the others will be in a database state conflict, making them not be recognized as valid and part of the current data center connected to it.
ソリューション
Restart all impacted nodes, so the cache on the nodes is cleared and the connection with the database is re-established.