How to clean up invalid, old, or stalled cluster nodes in Bamboo Data Center
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
This article describes how to clean up invalid, old, or stalled cluster nodes from the Bamboo Data Center configuration.
Environment
Bamboo Data Center in a clustered environment
Diagnosis
In a Bamboo cluster, all nodes' information are kept on the database and also on each node's filesystem. For integrity reasons, Bamboo does no automatically purges stale or old nodes. The cluster nodes listing can be found on the applications' Clustering menu along with the applications.xml file within a Support zip.
Under certain circumstances such as multiple tentatives to instantiate a new node, more than the expected number of cluster nodes may be listed. For example:
1
2
3
4
5
6
7
8
9
<cluster-information>
<node-count>4</node-count>
<nodes>
<node id="6b5971ed-1414-4fd8-882d-7606462b9c0a"/>
<node id="83b0e5af-dcb6-4858-b6f4-6cfa60c26fbd"/>
<node id="a31cd804-5d32-4afa-9c8a-356c4d88d577"/>
<node id="d912df75-387b-44fb-a672-e5c8c118f3fd" current-node="true"/>
</nodes>
</cluster-information>Cause
Whenever a new node joins a Bamboo cluster, a cluster-node.properties is created and added to the local <bamboo-home> folder of that node. That information is also added to the Bamboo database so other cluster nodes can know about any new members.
In the event of Bamboo node being decommissioned or added multiple times or by mistake in a cluster, the Database will keep the information and report such nodes as offline. Depending on how many times a node was tentatively added to a cluster, multiple offline nodes may appear as cluster members, which may not be ideal from a management perspective.
Solution
Bamboo 9.5 and later
In the Bamboo application, log in as an Administrator and go to:
Bamboo Administration → Overview → Clustering
Identify the nodes that are marked as OFFLINE
Validate if a node is really not required, sometimes nodes are just not running so they must be still considered as valid. You can check the Hostname field and compare their names.
If identical hostnames are found, log on to the Bamboo machine and validate its real
node.idfrom<bamboo-home>/cluster-node.propertiesDelete any old or invalid nodes by using the Actions menu
Bamboo 9.4 and earlier
As direct cluster nodes management over the UI was introduced only in Bamboo 9.5, stalled Bamboo cluster members cleanup should be done directly in the database.
Cleanup of stalled cluster nodes on the Database
On each valid Bamboo node, find the
<bamboo-home>/cluster-node.propertiesfile and take notes of thenode.idandnode.name(they usually match) - These are the nodes that you should keepStop Bamboo (all nodes)
Make sure you have a full backup of your <
Bamboo-Home>and<Bamboo-Install>folders, along with the DatabaseIn the Database, find the
cluster_node_heartbeattableDelete the rows from the
cluster_node_heartbeattable that don't match the node IDs you discovered on item #1Start Bamboo (both nodes)
Was this helpful?