Jira Data Center Health Check Failure: Unable to Write to Storage
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
All the Jira health checks fail intermittently with the ‘warning’: ‘Unable to write to <concerned healthcheck> storage.’
atlassian-jira.log
2025-02-07 07:44:05,935+1030 HealthCheck:thread-8 WARN [c.a.t.healthcheck.concurrent.SupportHealthCheckProcess] Health check 'Avatar Storage' failed with severity 'warning': 'Unable to write to avatar storage.'
2025-02-07 07:44:05,935+1030 HealthCheck:thread-8 ERROR [c.a.t.healthcheck.concurrent.SupportHealthCheckProcess] Health check 'Attachment Storage' failed with severity 'major': 'Unable to write to attachment storage.'
2025-02-07 07:44:05,935+1030 HealthCheck:thread-8 ERROR [c.a.t.healthcheck.concurrent.SupportHealthCheckProcess] Health check 'Backup Storage' failed with severity 'major': 'Can't write to backup storage.'
2025-02-07 07:44:05,935+1030 HealthCheck:thread-8 ERROR [c.a.t.healthcheck.concurrent.SupportHealthCheckProcess] Health check 'Backup and recovery' failed with severity 'critical': 'Parent directory '/mnt/jira/sharedhome/caches/indexesV2' of the index snapshot directory doesn't exist. This directory is not node specific, and lives in shared home. Check your system configuration.'When you access the Jira backend server directly, the issue is temporarily resolved. However, it reoccurs after some time.
Diagnosis
Additionally, the logs show an issue with file system access, particularly with the shared_home directory mounted via NFS on a server with Kerberos Authentication:
atlassian-jira.log
2025-01-30 04:35:12,983+1030 HealthCheck:thread-2 WARN [c.a.d.f.impl.filesystem.FilesystemPathImpl] Failed to delete file /mnt/jira/sharedhome/data/attachments/healthcheck-acfb4b08-5027-4923-a06e-83a3304b2ed2
java.nio.file.AccessDeniedException: /mnt/jira/sharedhome/data/attachments/healthcheck-acfb4b08-5027-4923-a06e-83a3304b2ed2
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:248) Cause
The shared_home directory is becoming unavailable at times, and Jira loses its file access privileges to it. A couple of things to check with the mount point:
NFS Mount permission issues: Jira may face permission or access issues while trying to write to the shared home directory on the NFS mount, especially on the initial health check run. When the user SSHs into the server and refreshes, it might be that their SSH session or actions (like refreshing the mount or triggering a refresh of the NFS mount status) temporarily resolve the issue.
NFS Mount Stability or Caching: The NSF mounts do take time to become fully available or stable after a reboot or a network glitch. It could be that when Jira starts, the NFS mount is in an inconsistent state, but as soon as you access it directly, it triggers a re-mount and resolves the issue.
Solution
When the shared home directory is default mounted with Kerberos Authentication (Krb5), the Kerberos token will expire for the Jira user after a certain time. After setting the mount options to auth_sys, this will be resolved.
Further validate the following specifics:
Verify if there is any issue with the NFS mount point used for the shared_home directory and if any authentication protocols are in use.
Try moving the shared home directory to a temporary location to determine if the NFS share is indeed contributing to the issue and further narrow down the scope of the issue. Refer How to move the shared-home folder in Jira Data Center
Was this helpful?