Repositories are not accessible due to NoSuchRepositoryException in Bitbucket Server and Data Center
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Repositories are not accessible after upgrading Bitbucket server / underlying OS. In some cases, Bitbucket application may not be accessible.
Environment
Bitbucket Data Center 8.8.2, applicable for all Bitbucket-supported versions
OS upgrade
Diagnosis
From the logs, you can see below NoSuchRepositoryException in atlassian-bitbucket.log
YYYY-MM-DD 11:53:01,306 DEBUG [ssh-scm-request-handler] e35e299c903332bf @1719RKQx7kyx3x0 1azbgth XX.XX.XX.XX SSH - git-upload-pack '/<myproj>/<myrepo>.git' c.a.s.i.s.g.p.s.SshCommandExitHandler <Not started>: Threw the following exception:
com.atlassian.bitbucket.repository.NoSuchRepositoryException: There is no repository with ID ds/0/h/5ff270cabac21276cd05/r/1234
at com.atlassian.stash.internal.scm.git.mesh.DefaultErrorTranslator.translateNoSuchRepository(DefaultErrorTranslator.java:379)
at com.atlassian.stash.internal.scm.git.mesh.DefaultErrorTranslator.translate(DefaultErrorTranslator.java:92)
at com.atlassian.stash.internal.scm.git.mesh.DefaultErrorTranslator.translateIfKnownCause(DefaultErrorTranslator.java:269)
at com.atlassian.stash.internal.scm.git.mesh.DefaultErrorTranslator.maybeTranslate(DefaultErrorTranslator.java:57)
at com.atlassian.stash.internal.scm.git.mesh.AbstractFutureResponseObserver.maybeTranslate(AbstractFutureResponseObserver.java:209)
at com.atlassian.stash.internal.scm.git.mesh.AbstractHostingFragmentResponseObserver.onError(AbstractHostingFragmentResponseObserver.java:107)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:479)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.atlassian.stash.internal.scm.git.mesh.LastSeenClientInterceptor$LastSeenClientListener.onClose(LastSeenClientInterceptor.java:40)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.atlassian.stash.internal.scm.git.mesh.StatefulClientCallListener.onClose(StatefulClientCallListener.java:34)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at com.atlassian.stash.internal.scm.git.mesh.DeadlinePropagatingClientInterceptor$DeadlinePropagatingListener.onClose(DeadlinePropagatingClientInterceptor.java:156)
at com.atlassian.stash.internal.scm.git.mesh.ErrorHandlingClientInterceptor$ErrorHandlingCall$1.onClose(ErrorHandlingClientInterceptor.java:149)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
... 1 frame trimmed
Suppressed: com.atlassian.bitbucket.scm.CommandFailedException: 'Unknown' could not be started
at com.atlassian.stash.internal.scm.git.mesh.AbstractHostingFragmentResponseObserver.ensureStarted(AbstractHostingFragmentResponseObserver.java:187)
at com.atlassian.stash.internal.scm.git.mesh.AbstractHostingFragmentResponseObserver.finish(AbstractHostingFragmentResponseObserver.java:201)
at com.atlassian.stash.internal.scm.git.mesh.AbstractHostingFragmentResponseObserver.onError(AbstractHostingFragmentResponseObserver.java:112)
... 23 common frames omittedCause
The possible cause of the issue is that the NFS mount where BITBUCKET_HOME/shared directory is mounted is missing/ unmounted on the server after upgrading OS / Bitbucket. Bitbucket cannot find the repositories located in BITBUCKET_HOME/shared directory and writes NoSuchRepositoryException in logs.
Solution
Verify if the NFS shared directory is mounted on Bitbucket nodes by running the below command
df -hPVerify if the nfs mount details are present in /etc/fstab
cat /etc/fstabℹ️ Please make sure the NFS mount options are as per Atlassian recommendations.
Recommended NFS mount options
rw,nfsvers=3,lookupcache=pos,noatime,intr,rsize=32768,wsize=32768,_netdevMount the NFS share if not mounted using
mountcommandmount -t nfs <NFS_SERVER_IP>:<directory_path_on_the_nfs_server> <path_on_bitbucket_node>
Was this helpful?