Bitbucket Data Center showing error 'SpringMVC dispatcher could not be started' while starting up

Problem

Bitbucket Server fails to startup.

The following message is displayed on the UI:

Cause

There are different root causes for this problem.

1. Cause #1:

   The following exception is reported in the atlassian-bitbucket.log file:

   catalina.out

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 15:08:16,767 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[bitbucket.application] - openFile(/var/atlassian/application-data/bitbucket/log/atlassian-bitbucket.log,true) call failed. java.io.FileNotFoundException: /var/atlassian/application-data/bitbucket/log/atlassian-bitbucket.log (Permission denied) at java.io.FileNotFoundException: /var/atlassian/application-data/bitbucket/log/atlassian-bitbucket.log (Permission denied) at at java.io.FileOutputStream.open(Native Method) at at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at at ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:28) at at ch.qos.logback.core.FileAppender.openFile(FileAppender.java:149) at at ch.qos.logback.core.FileAppender.start(FileAppender.java:108) at at ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:86) at at ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:96) at at ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:317) at at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:196) at at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:182) at at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62) at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149) at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135) at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99) at at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49) at at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75) at at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:148) at at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85) at at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55) at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:129) at at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:108) at at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:302) at at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:276) at at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288) at at com.atlassian.bitbucket.internal.lifecycle.DefaultStartupManager.<clinit>(DefaultStartupManager.java:18) at at com.atlassian.bitbucket.internal.spring.Bitbucket ServerWebApplicationInitializer.onStartup(Bitbucket ServerWebApplicationInitializer.java:65) at at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:175) at at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5423) at at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559) at at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549) at at java.util.concurrent.FutureTask.run(FutureTask.java:262) at at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at at java.lang.Thread.run(Thread.java:744)

   In this case, /var/atlassian/application-data/bitbucket/, which is being used as Bitbucket Server Home Directory, is not accessible to the user running Bitbucket Server. The permissions on the directories contained within <Bitbucket home directory> might be mixed where the user running Bitbucket Server is not the same.

   For further steps on how to diagnose this please see - How to View Detailed permissions for Bitbucket Server directory

2. Cause #2:

   The following exception is reported in the atlassian-bitbucket.log file:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 2018-07-08 01:01:11,128 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms. at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:667) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at java.lang.Thread.run(Thread.java:748) ... 18 frames trimmed Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms. at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628) ... 5 common frames omitted Caused by: liquibase.exception.DatabaseException: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms. at liquibase.database.jvm.JdbcConnection.getDatabaseProductName(JdbcConnection.java:43) at liquibase.database.core.OracleDatabase.isCorrectDatabaseImplementation(OracleDatabase.java:210) at liquibase.database.DatabaseFactory.findCorrectDatabaseImplementation(DatabaseFactory.java:111) at liquibase.integration.spring.SpringLiquibase.createDatabase(SpringLiquibase.java:471) at liquibase.database.ExtendedSpringLiquibase.createDatabase(ExtendedSpringLiquibase.java:40) at liquibase.integration.spring.SpringLiquibase.createLiquibase(SpringLiquibase.java:437) at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:32) ... 5 common frames omitted Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:602) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:195) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:145) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:85) at com.atlassian.stash.internal.hikari.ExtendedHikariDataSource.getConnection(ExtendedHikariDataSource.java:125) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:403) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:376) at liquibase.database.jvm.JdbcConnection.getDatabaseProductName(JdbcConnection.java:41) ... 11 common frames omitted

   In this case, the database used by Bitbucket Server is unavailable.

3. Cause#3

   The following exception is reported in atlassian-bitbucket.log

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 WARN [spring-startup] o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Cannot resolve reference to bean 'licenseHelper' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String 2021-07-30 14:52:40,809 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket 2021-07-30 14:52:40,818 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Cannot resolve reference to bean 'licenseHelper' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:723) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at java.lang.Thread.run(Thread.java:748) ... 18 frames trimmed Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sharedHomeLockAcquirer' defined in class path resource [stash-context.xml]: Cannot resolve reference to bean 'licenseHelper' while setting constructor argument; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:342) ... 5 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'licenseHelper' defined in URL [jar:file:/opt/atlassian/bitbucket/7.6.8/app/WEB-INF/lib/bitbucket-service-impl-7.6.8.jar!/com/atlassian/stash/internal/license/DefaultLicenseHelper.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798) ... 5 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796) ... 5 common frames omitted Caused by: java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.lang.String at liquibase.changelog.StandardChangeLogHistoryService.getRanChangeSets(StandardChangeLogHistoryService.java:324) at liquibase.changelog.AbstractChangeLogHistoryService.upgradeChecksums(AbstractChangeLogHistoryService.java:66) at liquibase.changelog.StandardChangeLogHistoryService.upgradeChecksums(StandardChangeLogHistoryService.java:293) at liquibase.Liquibase.checkLiquibaseTables(Liquibase.java:1221) at liquibase.Liquibase.update(Liquibase.java:193) at liquibase.Liquibase.update(Liquibase.java:179) at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:317) at liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:39) ... 5 common frames omitted

   This is caused by using an incompatible MySQL connector 8, whereas Bitbucket accepts only 5.1x

4. Cause#4

   The following exceptions are seen in atlassian-bitbucket.log

   1 2 3 4 5 6 7 2021-08-30 12:23:43,101 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'scheduledJobLifecycle' defined in URL [jar:file:/opt/atlassian/atlassian-bitbucket-6.5.1/app/WEB-INF/lib/bitbucket-platform-6.5.1.jar!/com/atlassian/stash/internal/scheduling/ScheduledJobLifecycle.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'instanceAnalyticsJob' defined in URL [jar:file:/opt/atlassian/atlassian-bitbucket-6.5.1/app/WEB-INF/lib/bitbucket-service-impl-6.5.1.jar!/com/atlassian/stash/internal/server/analytics/InstanceAnalyticsJob.class]: Unsatisfied dependency expressed through constructor parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'awsAnalyticsFactory' defined in URL [jar:file:/opt/atlassian/atlassian-bitbucket-6.5.1/app/WEB-INF/lib/bitbucket-service-impl-6.5.1.jar!/com/atlassian/stash/internal/server/analytics/DefaultAwsAnalyticsFactory.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.atlassian.stash.internal.server.analytics.DefaultAwsAnalyticsFactory]: Constructor threw exception; nested exception is java.lang.ExceptionInInitializerError Caused by: java.lang.IllegalArgumentException: Error in security property. Constraint unknown: jdk.disabled.namedCurves at sun.security.util.DisabledAlgorithmConstraints$Constraints.<init>(DisabledAlgorithmConstraints.java:329) at sun.security.util.DisabledAlgorithmConstraints.<init>(DisabledAlgorithmConstraints.java:100) at sun.security.ssl.SSLAlgorithmConstraints.<clinit>(SSLAlgorithmConstraints.java:50) ... 18 common frames omitted

   In this case, underlying Java JVM couldn't understand the parameter jdk.disabled.namedCurves and fails to start the service.

5. Cause #5

   The following exceptions are seen in atlassian-bitbucket.log

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 2023-01-11 02:31:02,377 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshHealthcheck' defined in URL [jar:file:/opt/atlassian/bitbucket/8.4.1/app/WEB-INF/lib/bitbucket-git-mesh-8.4.1.jar!/com/atlassian/stash/internal/scm/git/mesh/MeshHealthcheck.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshClient' parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Failed to start Mesh at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at java.lang.Thread.run(Thread.java:750) ... 18 frames trimmed Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshClient' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshClient' parameter 2; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Failed to start Mesh ... 6 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'routePropagatingClientInterceptor' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'routePropagatingClientInterceptor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Failed to start Mesh ... 6 common frames omitted Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'meshRouter' defined in com.atlassian.stash.internal.scm.git.GitWiring$MeshWiring: Unsatisfied dependency expressed through method 'meshRouter' parameter 5; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Failed to start Mesh ... 6 common frames omitted Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sidecarManager': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Failed to start Mesh at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) ... 5 common frames omitted Caused by: java.lang.IllegalStateException: Failed to start Mesh at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.startSidecar(DefaultSidecarManager.java:166) at com.atlassian.stash.internal.scm.git.mesh.DefaultSidecarManager.start(DefaultSidecarManager.java:109) ... 5 common frames omitted

Resolution

These are the resolutions for the root causes above:

1. Resolution for cause #1:

   Ensure the user has sufficient read/write/execute permissions on the above folder and restart the application.

   For the example above, the following commands were run:

   For non-Docker environment

   1 2 $ cd /var/atlassian/application-data/Bitbucket $ chown -R atlbitbucket:atlbitbucket *

   For Docker environment

   1 2 $ cd /var/atlassian/application-data/bitbucket $ chown -R bitbucket:bitbucket *

2. Resolution for cause #2:

   Make sure that the database service is up and running.

3. Resolution for cause#3:

   Download and install a compatible driver as described in Connecting Bitbucket to MySQL .

   Also, make sure that the driver in bitbucket.properties is of the form jdbc.driver=com.mysql.jdbc.Driverwhich is the syntax compatible with a 5.1x driver.

4. Resolution for cause #4:

   1. Verify if there is a problem with the version of ca-certificates java package.

   2. Use a different JDK version to start the Bitbucket server.

5. Resolution for cause #5:The error message states unsatisfied dependancy exception for GIT version on the Bitbucket server. On Some of the latest linux distributions GIT version is 2.38.x or later, which is still not supported on Bitbucket 8.6.x or below and will likely Bitbucket fails to start. Please refer Supported platform page for your Bitbucket version and compile a supported version of GIT to resolve the issue