Bitbucket Server Process Dies Unexpectedly Due to Linux OOM-Killer
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Bitbucket Server is installed on a Linux host.
The entire Bitbucket Server process suddenly terminates without warning. That is to say, the process ID (pid) is gone.
The browser shows a generic "cannot connect" or similar error, indicating that it is not able to reach the webpage.
Nothing out of the ordinary appears in the Bitbucket Server application logs (
$BITBUCKET_HOME/logs/atlassian-bitbucket.log), since the application was terminated without properly shutting down.Similarly, Tomcat logs (
<Bitbucket Server installation directory>/logs/catalina.out) also do not show errors.
As of Bitbucket Server 5.x+, no logs are written to the BITBUCKET_INSTALL directory. The contents of catalina.out will be written to BITBUCKET_HOME/log/atlassian-bitbucket.log file.
Diagnosis
On the host machine, look in the /var/log/ directory for the syslog or messages, and locate the timestamps spanning the approximate time when the pid was terminated. If you see entries similar to the following, then you know the process was a victim of the OOM-Killer:
Apr 8 11:29:48 bitbucket01 kernel: Out of memory: Kill process 1386 (java) score 358 or sacrifice
Apr 8 11:29:48 bitbucket01 kernel: Killed process 5388, UID 4048, (java) total-vm:1331564kB, anon
Apr 8 11:29:51 bitbucket01 kernel: java invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0,Cause
Bitbucket Server does not terminate its own pid unless a stop-bitbucket.sh is executed (which writes shutdown messages to the log). An unexpected pid termination clearly indicates the work of an external entity. Some examples include:
kill -9(user triggered or via script)Linux OOM-Killer
This KB article focuses on the Linux OOM-Killer, which is a feature on some Linux installations that will sacrifice processes to free up memory if the operating system experiences memory exhaustion for its own operations. Please note that this is different from Bitbucket Server running out of memory. In this case, the OS itself is in danger of running out of memory and thus starts terminating processes to avoid it.
Solution
Workaround
The way to mitigate this type of issue is to Add Swap Space to the system. This will keep Bitbucket from crashing while working on the resolution. This can be done live without having to stop Linux or Bitbucket. This will have performance impacts on Bitbucket so this should only be used to prevent a full outage.
Decide where on the disk there is available space. This example will put the swap space in the root user's home directory.
Create a file on the local disk. This is an example of 2G swap file:
dd if=/dev/zero of=/root/myswapfile bs=1M count=2048Change the permission of the swap file so that only root can access it.
chmod 600 /root/myswapfileMake and enable this file as a swap file using mkswap command.
mkswap /root/myswapfile swapon /root/myswapfileVerify whether the newly created swap area is available for your use.
swapon -s
Resolution
In the case of the OOM-Killer, the possible resolutions would be to:
Increase the amount of memory available on the host machine itself.
Decrease the amount of memory allocated to Bitbucket Server or competing processes on the machine.
Disable the OOM-Killer (not recommended).
For additional info on how the OOM-Killer operates, please see: http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/.
Was this helpful?