Troubleshooting "java.lang.OutOfMemoryError: unable to create new native thread" error in VPS installations

Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Symptoms

Fisheye cannot be started and the following appears in the atlassian-fisheye.log:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 2013-10-01 13:11:16,900 ERROR - Crucible Database setup failed. java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at com.jolbox.bonecp.ConnectionPartition.<init>(ConnectionPartition.java:236) at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:356) at com.jolbox.bonecp.provider.BoneCPConnectionProvider.createPool(BoneCPConnectionProvider.java:168) at com.jolbox.bonecp.provider.BoneCPConnectionProvider.configure(BoneCPConnectionProvider.java:142) at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:143) at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:84) at org.hibernate.cfg.SettingsFactory.createConnectionProvider(SettingsFactory.java:438) at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:91) at org.hibernate.cfg.Configuration.buildSettingsInternal(Configuration.java:2163) at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2159) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1383) at com.cenqua.crucible.hibernate.HibernateUtil.setUp(HibernateUtil.java:100) at com.cenqua.crucible.hibernate.DefaultDBControl.start(DefaultDBControl.java:243) at com.cenqua.crucible.hibernate.DefaultDBControl.start(DefaultDBControl.java:221) at com.cenqua.crucible.hibernate.DBControlFactoryImpl.<init>(DBControlFactoryImpl.java:40) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:108) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425) at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47) at org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:549) at org.mortbay.jetty.servlet.Context.startContext(Context.java:136) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at com.cenqua.fisheye.web.jetty.FishEyeWebApplicationContext.doStart(FishEyeWebApplicationContext.java:28) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at com.cenqua.fisheye.web.WebServer.start(WebServer.java:276) at com.cenqua.fisheye.ctl.Run.mainImpl(Run.java:233) at com.cenqua.fisheye.ctl.Run.main(Run.java:78) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.cenqua.fisheye.FishEyeCtl.mainImpl(FishEyeCtl.java:98) at com.cenqua.fisheye.FishEyeCtl.main(FishEyeCtl.java:41)  

Cause

Specifically, whatever user needs to run Fisheye needs to be able to open at least 200 files at a time, and be able to execute >100 processes at a time, since 100+ threads are created by Fisheye.

Basically the aproximated maximum number of threads of a JVM can be calculated this way:

(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads

Resolution

  1. Modify the limits.conf file with the following:

  2. sudo vim /etc/security/limits.conf

  3. Add the following for the user that runs FISHEYE.

    limits.conf

    1 2 3 4 5 6 7 8 9 10 11 12 13 #<domain> <type> <item> <value> # #* soft core 0 #root hard core 100000 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #ftp - chroot /ftp #@student - maxlogins 4 fisheye soft nofile 4096 fisheye hard nofile 8192 
  4. Modify the common-session file with the following:

    sudo vim /etc/pam.d/common-session

  5. Add the following line:

    common-session

    1 session required pam_limits.so
  6. Restart Fisheye.

Updated on April 15, 2025

Still need help?

The Atlassian Community is here for you.