"File corrupted while reading record" error while starting Bitbucket Server Mirror
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Getting the "File corrupted while reading record" error in the application logs while starting Bitbucket Mirror
Environment
Bitbucket Mirror version 7.21.7
Diagnosis
You'll see the below error message in the Bitbucket mirror application logs, indicating that the underlying H2 DB is corrupt:
1
2
3
ERROR [spring-startup] com.zaxxer.hikari.pool.HikariPool bitbucket - Exception during pool initialization.
org.H2.jdbc.JdbcSQLException: File corrupted while reading record: "index not found 483". Possible solution: use the recovery tool [90030-176]
at org.H2.message.DbException.getJdbcSQLException(DbException.java:344) If the mirror process or the server was forcibly shut down, it could cause the H2 process to fail to finish some operation that would result in disk corruption. This isn't unique to Bitbucket mirrors. Any piece of running software has a chance for this to happen in those kinds of circumstances.
A few additional points to note:
The nodes in a mirror farm do not share any directories or files with each other. Even though the directory name is
$BITBUCKET_HOME/sharedmentioned in Step #2 of Set up and configure a mirror farm, it should all be local and not shared.Bitbucket Data Center primary(upstream) and mirrors use the same codebase so the directory paths used for both are the same, e.g.
$BITBUCKET_HOME/shared/bitbucket.properties.However, mirror farm nodes do not actually share the
$BITBUCKET_HOME/shareddirectory and its contents. Each node should have its own copy.In the Install Bitbucket Data Center documentation for the upstream, there is a section for "Provision your shared database" and "Provision your shared file system". These are required for the upstream server.
On the other hand, the Set up and configure a mirror farm documentation does not have these steps.
Using the individual "shared" location approach should be ok. You can set up each mirror node to mount an NFS location to store its repositories, just each mirror node would need to have its own distinct location on NFS.
Please note that the entire
$BITBUCKET_HOMEshould not be stored on the network storage, only the$BITBUCKET_HOME/shared(even though it is not going to be shared).
The most common configuration is that each mirror node would use its own local storage. This local storage is faster than network storage but you do have to make sure that you have enough disk space for all of your mirrored repositories and the SCM cache that is built up.
Cause
Some of the possible causes of the issue are:
OS patching was done while the mirror was running and before the reboot.
Having the data on a shared NAS. Mirror farms don't require NFS and storing the data on it adds latency. It is mentioned in our configure a mirror farm guide:
Each node of a mirror farm is considered unique - While there is some internode communication in a mirror farm, nodes of a single mirror farm should not use shared storage for their home directory.
It is recommended that each node has its home directory on a local disk as opposed to having it on a shared NFS store.
In case there was some form of upgrade or activity done while the mirror was running that corrupted the H2 database.
In case there are multiple nodes being pointed to the same H2 because the NFS share was set up like you would if you were installing a full Bitbucket cluster rather than mirrors.
The most likely cause is a combination of an outside update/patch and running on an NFS share.
Solution
As the underlying H2 Database is corrupt, you'll need to rebuild the mirror from scratch. To do this, you'll need to follow these steps:
Stop the mirror
Delete the contents of the home directory on the mirror (except the
bitbucket.propertiesfile, which you can keep)Delete the mirror from the UI on the primary Data Center instance
Start the mirror and run through the setup wizard
Choose which projects to mirror
Once that's done the mirror should be operational again.
Ensure to remove the NFS share from the environment, have things run locally, and ensure the mirror is shut down prior to any outside processes that could interact with the system being run.
We have a backup of the mirror configuration with the Database. Can we restore from the backup?
Mirrors don't have a native backup process and we don't typically expect backups to be taken so we aren't sure about the results, but since it's read-only and won't affect the primary(upstream) server, there isn't a reason not to try to see if it is able to connect and self-heal the differences that have been pushed to the upstream server like it normally would.
If the backup was taken right before rebuilding the mirror, then the backup will have the same corrupt database, and restoring it would put us back to square one.
Our recommendation would be to restore from a backup if that backup was taken while the mirror was offline. Because of the self-healing ability of the mirrors, it is possible for any inconsistencies to be resolved.
Try to restore the H2 database and restart the mirror.
After shutting down the mirror, restore the entire drive that was backed up and confirm if everything comes back up without issues.
Was this helpful?