Bitbucket hits OOME when SBC is backing up sta_shared_lob table
Platform Notice: Data Center Only - This article only applies to Atlassian apps on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
Symptoms
When Bitbucket Server Backup Client is performing a backup, it hits to OOME and from the logs, the last table that gets backed up before the memory issues starts is sta_shared_lob.
2015-03-03 05:07:34,702 INFO [threadpool:thread-124] admin @851LERx306x483x2 10.4.0.121,127.0.0.1 "POST /mvc/admin/backups HTTP/1.0" c.a.s.i.b.l.DefaultLiquibaseMigrationDao Backing up sta_shared_lob tableℹ️ sta_shared_lob is a table that persists of large objects that's used by repository hook settings and user settings
So the 'getting started' data is part of the user settings (coloration with sta_user_settings table)
The table is used for repo (hook) configurations (coloration with sta_repo_hook table)
In this scenario, it may look like this table may contain a huge amount of data that causes OOME when it is loaded into memory.
Solution
At this stage, we would be interested to see how many rows are in this table and the total size
select pg_size_pretty(pg_relation_size('sta_shared_lob'));select count(1) from sta_shared_lob;
If the size is unreasonably huge, perhaps this would be the bug that you're hitting BSERV-7137 - Database backup can cause OutOfMemoryError for huge database tables (fixed in 3.10.0 and later)
Workaround
For the workaround, you would need to:
1. Identify the rows which do have not have reference from other tables.
select id
from sta_shared_lob
where not exists (select sta_repo_hook.lob_id from sta_repo_hook where sta_repo_hook.lob_id = sta_shared_lob.id)
and not exists (select sta_user_settings.lob_id from sta_user_settings where sta_user_settings.lob_id = sta_shared_lob.id)2. Remove those rows
delete from sta_shared_lob
where id in
(select id
from sta_shared_lob
where not exists (select sta_repo_hook.lob_id from sta_repo_hook where sta_repo_hook.lob_id = sta_shared_lob.id)
and not exists (select sta_user_settings.lob_id from sta_user_settings where sta_user_settings.lob_id = sta_shared_lob.id));ℹ️ The table sta_shared_lob has constraints that will stop deletion if data are present on another table
⚠️ Please backup your $BITBUCKET_HOME and the database first before performing any altering it
Resolution
The resolution is to watch this JIRA ticket until the fix has been applied BSERV-7137 - Database backup can cause OutOfMemoryError for huge database tables (fixed in 3.10.0 and later)
Was this helpful?