High CPU Usage caused by Ancestors Check in Confluence
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Problem
Corrupted entries in the application Ancestors Table (responsible for maintaining the relationship in between parent and descendant pages on Confluence) can cause View Page threads to run for a long period, triggering symptoms like High CPU Usage and intense GC Activity, which can eventually lead the application to an outage. Thread Dumps will usually show that the offending threads are executing the getInheritedContentPermissionSets (accountable for checking permissions) method being executed:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
"http-nio-8090-exec-20 uri:/pages/viewpage.action username:" #274 daemon prio=5 os_prio=0 tid=0x00007f6484011000 nid=0x2e8b runnable [0x00007f61c350f000]
java.lang.Thread.State: RUNNABLE
at net.sf.hibernate.impl.AbstractVisitor.processValue(AbstractVisitor.java:67)
at net.sf.hibernate.impl.AbstractVisitor.processValues(AbstractVisitor.java:36)
at net.sf.hibernate.impl.SessionImpl.isUpdateNecessary(SessionImpl.java:2666)
at net.sf.hibernate.impl.SessionImpl.flushEntity(SessionImpl.java:2581)
at net.sf.hibernate.impl.SessionImpl.flushEntities(SessionImpl.java:2513)
at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2308)
at net.sf.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1831)
at net.sf.hibernate.impl.SessionImpl.getQueries(SessionImpl.java:1594)
at net.sf.hibernate.impl.SessionImpl.find(SessionImpl.java:1559)
at net.sf.hibernate.impl.QueryImpl.list(QueryImpl.java:49)
at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao.lambda$getInheritedContentPermissionSets$0(HibernateContentPermissionSetDao.java:38)
at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao$$Lambda$766/775792763.doInHibernate(Unknown Source)
at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
at org.springframework.orm.hibernate.HibernateTemplate.executeFind(HibernateTemplate.java:341)
at com.atlassian.confluence.security.persistence.dao.hibernate.HibernateContentPermissionSetDao.getInheritedContentPermissionSets(HibernateContentPermissionSetDao.java:25)Diagnosis
This problem is triggered regardless of environmental variables, but its symptoms are usually the following:
High CPU Usage.
Garbage Collection Threads being heavily used.
Also, it is important to diagnose if there are corrupted entries in the Ancestors Table or not, since those are known to cause this issue.
Is my Ancestors Table corrupted?
In order to diagnose if there are indeed corrupted entries in the Ancestors Table, an administrator may run the query below at Confluence's database:
Is my Ancestors Table corrupted?
1
2
3
4
5
SELECT A.ANCESTORID, A.DESCENDENTID, COUNT(A.DESCENDENTID)
FROM CONFANCESTORS A, CONTENT B
WHERE B.CONTENTID = A.DESCENDENTID
GROUP BY A.ANCESTORID, A.DESCENDENTID HAVING COUNT(A.DESCENDENTID) != 1
ORDER BY 3 DESCIn case it has returned any results, then it is corrupted indeed.
Cause
This is caused by corrupted entries in the application Ancestors Table - Responsible for maintaining the relationship in between parent and descendant pages on Confluence.
Solution
Workaround
The workaround consists of re-constructing the application Ancestors Table as described in Rebuilding the Ancestor Table documentation.
Resolution
There have been some suggestions to properly log and alert administrators when there are problems with the Ancestors Table, being the main one implemented already:
With proper logging, administrators can proactively rebuild the table upon need, however, this doesn't fix the cause behind why the table has been corrupted in first place and this is still pending a fix. Actions can be tracked using below bugs:
Was this helpful?