Bitbucket fails to start in disaster recovery mode.
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Problem
The following issues are observed in Bitbucket when trying to start in disaster recovery:
Bitbucket fails to start
The following appears in the log:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
2020-10-27 13:56:55,571 WARN [spring-startup] o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'fullIntegrityCheckStartupTask'; nested exception is com.atlassian.bitbucket.DataStoreException: A database error has occurred.
2020-10-27 13:56:55,764 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /nfs/bitbucket
2020-10-27 13:56:55,774 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /home/atlassian/bitbucket/application_data
2020-10-27 13:56:55,781 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.context.ApplicationContextException: Failed to start bean 'fullIntegrityCheckStartupTask'; nested exception is com.atlassian.bitbucket.DataStoreException: A database error has occurred.
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:174)
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:122)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:894)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.lang.Thread.run(Thread.java:748)
... 8 frames trimmed
Caused by: com.atlassian.bitbucket.DataStoreException: A database error has occurred.
at com.atlassian.stash.internal.aop.ExceptionRewriteAdvice.afterThrowing(ExceptionRewriteAdvice.java:47)
at com.atlassian.stash.internal.integrity.FullIntegrityCheckHelper.scheduleOpenPullRequestChecksByRescope(FullIntegrityCheckHelper.java:58)
at com.atlassian.stash.internal.integrity.FullIntegrityCheckHelper$$FastClassBySpringCGLIB$$aec74624.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at com.atlassian.stash.internal.integrity.FullIntegrityCheckHelper$$EnhancerBySpringCGLIB$$b59d03bf.scheduleOpenPullRequestChecksByRescope(<generated>)
at com.atlassian.stash.internal.integrity.FullIntegrityCheckStartupTask.lambda$start$0(FullIntegrityCheckStartupTask.java:81)
at com.atlassian.stash.internal.user.DefaultEscalatedSecurityContext.call(DefaultEscalatedSecurityContext.java:59)
at com.atlassian.stash.internal.integrity.FullIntegrityCheckStartupTask.start(FullIntegrityCheckStartupTask.java:78)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:182)
... 10 common frames omitted
Caused by: org.hibernate.NonUniqueResultException: query did not return a unique result: 2
at org.hibernate.query.internal.AbstractProducedQuery.uniqueElement(AbstractProducedQuery.java:1597)
at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:392)
at com.atlassian.stash.internal.crowd.HibernateUserDao.internalFindByName(HibernateUserDao.java:452)
at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:131)
at com.atlassian.stash.internal.crowd.HibernateUserDao.findByName(HibernateUserDao.java:72)
at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:167)
at com.atlassian.crowd.directory.AbstractInternalDirectory.findUserByName(AbstractInternalDirectory.java:64)
at com.atlassian.crowd.directory.AbstractForwardingDirectory.findUserByName(AbstractForwardingDirectory.java:74)
at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.findUserByName(DirectoryManagerGeneric.java:315)
at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.findUserByName(ApplicationServiceGeneric.java:307)
at com.atlassian.crowd.embedded.core.CrowdServiceImpl.getUser(CrowdServiceImpl.java:86)
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.getCrowdUser(CrowdAwareLoadListener.java:130)
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.setCrowdUserIfNecessary(CrowdAwareLoadListener.java:149)
at com.atlassian.stash.internal.hibernate.CrowdAwareLoadListener.onPostLoad(CrowdAwareLoadListener.java:87)
at org.hibernate.engine.internal.TwoPhaseLoad.postLoad(TwoPhaseLoad.java:517)
at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.postLoad(AbstractRowReader.java:309)Cause
This error is caused by a duplicate value in the database. This might be due to:
Some corruption in the directory cache.
Corruption in the database.
Diagnosis
Find duplicated users in the cwd_user table
Diagnosis Query - 1: Find duplicate users in the cwd_user table that has the same External ID and belong to the same directory:
1
2
3
4
SELECT directory_id, external_id
FROM cwd_user
GROUP BY directory_id, external_id
HAVING ( COUNT(external_id) > 1)Diagnosis Query - 2: Find duplicate users in the cwd_user table that has the same lower_user_name and belong to the same directory:
1
2
3
4
SELECT lower_user_name, directory_id
from cwd_user
GROUP BY lower_user_name, directory_id
HAVING (COUNT(lower_user_name) > 1);Resolution
Deleting the user from the database
Always backup Bitbucket before removing or modifying any data on the database and ensure Bitbucket is shut down.
Step 1: Find the duplicated user IDs
Run the query below to get the users IDs of the duplicated user(s):
1
SELECT id, lower_user_name FROM cwd_user WHERE external_id = '<duplicated external_id from diagnostic query>' or
1
SELECT id, lower_user_name FROM cwd_user WHERE lower_user_name = '<duplicated username from diagnostic query>' The result should be as follow:
id | user_name |
|---|---|
229377 | admin |
229378 | admin |
Note the IDs provided in the query above. Determine which entry to delete in the next step (Step 2).
Step 2: Determine which IDs to delete by checking if any of the duplicated users currently belong to any group
Run the following SQL query against the duplicated IDs result. The SQL query will yield the number of groups that each duplicated_user ID is tied to:
1
SELECT count(*), child_user_id FROM cwd_membership WHERE child_user_id in (<duplicated_id_1>,<duplicated_id_2>) GROUP BY child_user_idStep 3.1 If one of the duplicated ID does not return any group membership result
If the SQL query above return with something as follows,
count(*) | child_user_id |
|---|---|
2 | 229377 |
Proceed with safely deleting the other duplicated user id (not the one shown from the SQL query):
1
2
3
DELETE FROM cwd_user_attribute WHERE user_id = <duplicated_id>;
DELETE FROM cwd_user_credential_record WHERE user_id = <duplicated_id>;
DELETE FROM cwd_user WHERE id = <duplicated_id>;Step 3.2: If both of the duplicated ID returns a number of group memberships
If the SQL query in step 2 return with something as follows:
count(*) | child_user_id |
|---|---|
2 | 229377 |
2 | 229378 |
Run the following SQL query to obtain the duplicated users group membership:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
SELECT
u.user_name,
g.id as GroupID,
g.group_name
FROM
cwd_user u
JOIN cwd_membership m
ON
u.id = m.child_user_id
JOIN cwd_group g
ON
g.id = parent_id
WHERE u.user_name = '<duplicated username>'
ORDER BY 2 DESC, 1 ASC;Take note of the result of the above SQL query. Proceed with safely deleting one of the duplicated user ids. Run the below queries in order:
1
2
3
4
DELETE FROM cwd_membership WHERE child_user_id = <duplicated_id>;
DELETE FROM cwd_user_attribute WHERE user_id = <duplicated_id>;
DELETE FROM cwd_user_credential_record WHERE user_id = <duplicated_id>;
DELETE FROM cwd_user WHERE id = <duplicated_id>;Once you have completed the above steps, attempt to start Bitbucket to see if the issue has been resolved.
If you continue to experience issues, please contact support.atlassian.com/contact for further help.
Was this helpful?