Node encounters OutOfMemory with hundreds of "Asking if operation execution has been started" WARN messages
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
Hundreds of WARN messages like the following are in the atlassian-bitbucket-yyyy-mm-dd.log:
1
2018-04-18 07:18:00,882 WARN [cached12] c.h.s.i.o.impl.Invocation [10.64.98.221]:5701 [stash-prod-cluster] [3.5.2-atlassian-37] Asking if operation execution has been started: com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService$InvokeIsStillRunningOperationRunnable@336f67ee
These can lead to OutOfMemory ERRORS that can bring down nodes.
Cause
This is caused by Hazelcast bug 5468 which is described in Hazelcast Cluster members going out of memory due to huge number of “IsStillRunningService” objects. Bitbucket 4.14.x uses 3.5.2 of the Hazelcast library which is based on Hazelcast 3.5 and contains Hazelcast bug 5468. This bug has been fixed in Hazelcast 3.7.x.
Resolution
Upgrade to Bitbucket 5.x versions which use Hazelcast 3.7.4.
Was this helpful?