Confluence での Ancestors Check による CPU 使用率の増加
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
問題
Corrupted entries in the application Ancestors Table (responsible for maintaining the relationship in between parent and descendant pages on Confluence) can cause View Page threads to run for a long period, triggering symptoms like High CPU Usage and intense GC Activity, which can eventually lead the application to an outage. Thread Dumps will usually show that the offending threads are executing the getInheritedContentPermissionSets (accountable for checking permissions) method being executed:
"http-nio-8090-exec-20 uri:/pages/viewpage.action username:" #274 daemon prio=5 os_prio=0 tid=0x00007f6484011000 nid=0x2e8b runnable [0x00007f61c350f000]
java.lang.Thread.State: RUNNABLE
at net.sf.hibernate.impl.AbstractVisitor.processValue(AbstractVisitor.java:67)
at net.sf.hibernate.impl.AbstractVisitor.processValues(AbstractVisitor.java:36)
at net.sf.hibernate.impl.SessionImpl.isUpdateNecessary(SessionImpl.java:2666)
at net.sf.hibernate.impl.SessionImpl.flushEntity(SessionImpl.java:2581)
at net.sf.hibernate.impl.SessionImpl.flushEntities(SessionImpl.java:2513)
at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2308)
at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1831)
at net.sf.hibernate.impl.SessionImpl.getQueries(SessionImpl.java:1594)
at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1559)
at net.sf.hibernate.impl.QueryImpl.list(QueryImpl.java:49)
at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao.lambda$getInheritedContentPermissionSets$0(HibernateContentPermissionSetDao.java:38)
at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao$$Lambda$766/775792763.doInHibernate(Unknown Source)
at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
at org.springframework.orm.hibernate.HibernateTemplate.executeFind(HibernateTemplate.java:341)
at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao.getInheritedContentPermissionSets(HibernateContentPermissionSetDao.java:25)
診断
This problem is triggered regardless of environmental variables, but its symptoms are usually the following:
- High CPU Usage.
- Garbage Collection Threads being heavily used.
Also, it is important to diagnose if there are corrupted entries in the Ancestors Table or not, since those are known to cause this issue.
Is my Ancestors Table corrupted?
In order to diagnose if there are indeed corrupted entries in the Ancestors Table, an administrator may run the query below at Confluence's database:
SELECT A.ANCESTORID, A.DESCENDENTID, COUNT(A.DESCENDENTID)
FROM CONFANCESTORS A, CONTENT B
WHERE B.CONTENTID = A.DESCENDENTID
GROUP BY A.ANCESTORID, A.DESCENDENTID HAVING COUNT(A.DESCENDENTID) != 1
ORDER BY 3 DESC
In case it has returned any results, then it is corrupted indeed.
原因
This is caused by corrupted entries in the application Ancestors Table - Responsible for maintaining the relationship in between parent and descendant pages on Confluence.
回避策
The workaround consists of re-constructing the application Ancestors Table as described in Rebuilding the Ancestor Table documentation.
ソリューション
There have been some suggestions to properly log and alert administrators when there are problems with the Ancestors Table, being the main one implemented already:
With proper logging, administrators can proactively rebuild the table upon need, however, this doesn't fix the cause behind why the table has been corrupted in first place and this is still pending a fix. Actions can be tracked using below bugs:
- CONFSERVER-55515 - High CPU Usage caused by Ancestors Check when CONFANCESTORS table is corrupted Closed
- CONFSERVER-53054 - Confluence Questions causes certain functions to be very slow Gathering Impact
- CONFSERVER-54945 - CONFANCESTORS table obtains duplicate record after a page page re-order to space root Closed
- CONFSERVER-52736 - Ancestor table is broken after reordering pages and applying restrictions Closed