Jira server process is unexpectedly terminated in Linux

Summary

The Jira application is being terminated unexpectedly in a Linux environment. When using Jira Data Center, this issue might occur on a specific node only.

Diagnosis

• When checking the Jira application logs <JIRA-HOME>/log/atlassian-jira.log, we can see that the Jira process is being terminated unexpectedly. There is no clean shutdown process, and we see a big time gap in the logs:

  1 2 3 4 5 6 7 8 9 10 11 2021-10-26 01:49:54,218+0300 http-nio-8051-exec-154 url:/rest/dev-status/1.0/issue/summary username:XXXXX WARN XXXXX 109x8773607x4 1ycojow XXX.XXX.XX.XX,XX.XXX.XX.XXX /rest/dev-status/1.0/issue/summary [c.a.j.p.devstatus.provider.DefaultCoordinator] fisheye4 (not_configured) 2021-10-26 03:59:09,185+0300 localhost-startStop-1 INFO [c.a.jira.startup.JiraHomeStartupCheck] The jira.home directory '/opt/eejira/datacenter/jira.home' is validated and locked for exclusive use by this instance. 2021-10-26 03:59:09,220+0300 localhost-startStop-1 INFO [c.a.jira.startup.JiraStartupLogger] **************** Jira starting... **************** 2021-10-26 03:59:09,272+0300 localhost-startStop-1 INFO [c.a.jira.startup.JiraStartupLogger] ___ Environment _____________________________

• The same gap can be observed in the Tomcat Logs in <JIRA_INSTALL>/logs/catalina.out

• The following appears in the /var/log/messages, /var/log/syslog, or the systemd kernel journal of the Linux server where the Jira application was terminated:

  1 2 3 4 5 6 7 8 9 10 11 12 13 Oct 26 01:49:56 esjirp82 systemd: Removed slice User Slice of root. Oct 26 01:49:57 esjirp82 kernel: Disabling lock debugging due to kernel taint Oct 26 01:49:57 esjirp82 kernel: mce: [Hardware Error]: Machine check events logged Oct 26 01:49:57 esjirp82 kernel: mce: Uncorrected hardware memory error in user-access at fef7d0cc0 Oct 26 01:49:57 esjirp82 kernel: MCE 0xfef7d0: Killing java:51745 due to hardware memory corruption Oct 26 01:49:57 esjirp82 kernel: MCE 0xfef7d0: dirty LRU page recovery: Recovered Oct 26 01:49:57 esjirp82 kernel: MCE: Killing http-nio-8051-e:22979 due to hardware memory corruption fault at 14935d0000 Oct 26 01:49:57 esjirp82 kernel: MCE: Killing http-nio-8051-e:22979 due to hardware memory corruption fault at 14935d0000 Oct 26 01:49:57 esjirp82 kernel: MCE: Killing http-nio-8051-e:22979 due to hardware memory corruption fault at 14935d0000 Oct 26 01:49:57 esjirp82 kernel: MCE: Killing http-nio-8051-e:22979 due to hardware memory corruption fault at 14935d0000 Oct 26 01:49:57 esjirp82 kernel: MCE: Killing http-nio-8051-e:22979 due to hardware memory corruption fault at 14935d0000 Oct 26 01:49:57 esjirp82 kernel: MCE: Killing http-nio-8051-e:22979 due to hardware memory corruption fault at 14935d0000 Oct 26 01:49:57 esjirp82 abrt-hook-ccpp: Process 51745 (java) of user 1001 killed by SIGABRT - dumping core

Cause

This problem usually occurs when there is a hardware memory issue on the Linux server. As mentioned in this HP support article:

Solution

Since this issue is specific to the Linux server outside of the Jira application control, we recommend you to reach out to your Linux OS Administrator and share with them the error found in the /var/log/syslog file.