Inodes utilization on a NFS-mounted BITBUCKET_HOME/shared is high with Bitbucket Data Center

Summary

The <BITBUCKET_HOME>/shared directory is mounted on a NFS volume. Monitoring tools show very large inodes usage, for example more than 80% of inodes are already used. Obviously, this is not a healthy state and it needs investigation.

Environment

The solution has been validated in Bitbucket 8.19.8 but may be applicable to other versions.

This only affects Bitbucket running with a NFS-mounted <BITBUCKET_HOME>/shared directory.

Diagnosis

Health monitoring tools may report large inodes utilization, or small number of inodes available. It is also possible that manual checking of the mounted filesystems revealed large inodes allocation.

It is important to know that, although the alert about "more than X% of inodes used" may sound alarming, one should check the absolute number of still available inodes - it can be that there are millions available.

• You can manually check the status of inodes - execute the following command on the Linux server that is running Bitbucket. This command will output the number of used and available inodes for all mounted file systems; check the one corresponding to <BITBUCKET_HOME>/shared directory:

  1 df -i

• To get more information about which directories are using most of the inodes, you can execute the following command on the same Linux server, the one where Bitbucket runs. This command can run for a long time, so be sure it won't interfere with regular operations. Output of this command will be the number of inodes used by files stored in the <BITBUCKET_HOME>/shared directory:

  1 2 3 4 cd <BITBUCKET_HOME>/shared du -sc --inodes */*

Cause and solution

Numbers returned by "df -i" and "du -sc --inodes" are similar

If numbers returned by df -i and du -sc --inodes commands are similar, all inodes are used by files stored in <BITBUCKET_HOME>/shared directory.

Since that directory holds all Git repository data, you must not delete anything inside it.

To proceed, please raise a support ticket with us.

Numbers returned by "df -i" and "du -sc --inodes" differ significantly

However, if numbers returned by two commands are not even close - the one reported by "df -i" being larger than the other - it is a sign that inodes are mostly consumed by files on the NFS server..

For example, it is possible that on the NFS server's side, the block device containing NFS volume mounted at <BITBUCKET_HOME>/shared is not used exclusively for Bitbucket. What's more, it can be shared with other NFS exports, or it can be one large block device which is only partially - some directories only - expored via NFS and used for other purposes too.

To check, you can use these commands on the NFS server - they will show all directories exported through NFS, and status of inodes for all block devices. By comparing the two one can determine what's the state of NFS server's exports and volumes:

If NFS-expored directory that is mounted at <BITBUCKET_HOME>/shared is indeed shared with other software or NFS exports, administrators of the NFS server must check that and clean unnecessary files or solve it in some other way.