Confluence throws 500 error due to lock timeout in VCacheLock

Summary

Problem

Sometimes Confluence may respond with 500x error pages, have a general performance issue, or even a full outage, due to the lock timeout in VCacheLock which is used by I18NBean.

Either of the following appears in the atlassian-confluence.log:

or:

Symptoms

The system becomes unresponsive. Thread dumps show HTTP threads hung waiting for the lock. Thread dumps do not show what threads owns the lock. After a heap dump investigation, it will be clear that lock owner is either dead or spending time in an unrelated piece of code.

The lock has been 'leaked' somehow. Threads are blocking on it, but it is never going to be unlocked. Once the system enters this state, it can only be healed by a restart.

Diagnosis

Environment

• Confluence Server

• Confluence Data Center

• Brikit Themepress 1.x through 2.2.x installed

Diagnostic Steps

• Capture Thread Dumps while the issue is occurring

• First, check the status of all thread states across the thread dumps, using the following:

  1 $ grep -A1 '-exec-' confluence_threads.* | grep State | sort | uniq -c

  • If the result looks like this, with all threads in a WAITING (parking) state, you may have this issue and should follow the second grep below:

    1 2 3 4 5 6 7 $ grep -A1 '-exec-' confluence_threads.* | grep State | sort | uniq -c 300 confluence_threads.1535641226.txt- java.lang.Thread.State: WAITING (parking) 300 confluence_threads.1535641241.txt- java.lang.Thread.State: WAITING (parking) 300 confluence_threads.1535641255.txt- java.lang.Thread.State: WAITING (parking) 300 confluence_threads.1535641270.txt- java.lang.Thread.State: WAITING (parking) 300 confluence_threads.1535641285.txt- java.lang.Thread.State: WAITING (parking) 300 confluence_threads.1535641301.txt- java.lang.Thread.State: WAITING (parking)

• Second, grep each thread dump using the following method to look for the lock:

  1 $ cat thread_dump.txt | egrep "(MemoryStore.put|MemoryStore.get)" -B 10

  • If the results match this example, you have likely run into this issue:

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 $ cat confluence_threads.1535641285.txt | egrep "(MemoryStore.put|MemoryStore.get)" -B 1 java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00007f1e048cd408> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.put(SelectableConcurrentHashMap.java:641) at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.put(SelectableConcurrentHashMap.java:383) at net.sf.ehcache.store.MemoryStore.put(MemoryStore.java:283) -- java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00007f1e048cd408> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at net.sf.ehcache.store.chm.SelectableConcurrentHashMap$Segment.get(SelectableConcurrentHashMap.java:721) at net.sf.ehcache.store.chm.SelectableConcurrentHashMap.get(SelectableConcurrentHashMap.java:324) at net.sf.ehcache.store.MemoryStore.get(MemoryStore.java:344)

• For full confirmation you have run into this issue, contact Atlassian Support.

Cause

This happens when thread-X and thread-Y both lock the cache, then other threads try to lock the cache also and must for 30 seconds, then an exception occurs.

Multiple third party vendors have been identified with plugins that can run into this issue. Below are some known vendors which have fixes to close this issue:

Solution

Workaround

Capture thread dumps and restart Confluence to recover the system.

• Disable the Brikit Themepress plugin until a fixed version is available

• If you are unable to disable the Brikit Themepress plugin via the user interface, you may need to delete the plugin from the database

• Contact Brikit Support for a patched version of Themepress, until a fixed version is available

Resolution

Brikit's bugs linked above note the issue should be resolved in Themepress 2.3.x.

Upgrading Themepress to the fixed version should prevent the issue from occurring again.