Bamboo Elastic Agents fail to start up or get stuck in a pending state.
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
The steps outlined on this article are provided AS-IS. This means we've had reports of them working for some customers — under certain circumstances — yet are not officially supported, nor can we guarantee they'll work for your specific scenario.
You may follow through and validate them on your own non-prod environments prior to production or fall back to supported alternatives if they don't work out.
We also invite you to reach out to our Community for matters that fall beyond Atlassian's scope of support!
Bamboo Elastic Agents get stuck in a pending state when triggered or started manually. Additionally, there are scenarios where the EC2 instance fails to reach a running state, displaying the error message, "Elastic Bamboo Error: EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXXXXXXXX failed."
Environment
This solution has been tested in the following environments, 9.2.7, 9.2.14, 9.3.4 and 9.4.3 but the solution will be applicable for other Bamboo versions as well.
Diagnosis
As mentioned earlier, there are two alternative scenarios:
Scenario 1
Bamboo elastic agents get stuck in a pending state for a long time. View the instance using ,select
> Elastic Bamboo > Instances. This shows the below as the state of the instance:
Scenario 2
The EC2 instance fails to start with the error message below which can be seen in the <BAMBOO-HOME>/logs/atlassian-bamboo.log
1
2
3
Elastic Bamboo Error : EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXXXXXXXXX failed.
(com.amazonaws.services.ec2.model.AmazonEC2Exception : You are not authorized to perform this operation. User: arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME} is not authorized to perform: ec2:RunInstances on resource: arn:aws:ec2:{region}:{ACCOUNT_ID}:{RESOURCE_TYPE}/instance/* with an explicit deny in a service control policy. Encoded authorization failure message: XXXXXXXXXXXXX
....After attempting to decode the Encoded authorization failure as shown below, we can see a failure due to an AWS Service Control Policy.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ export AWS_MESG=XXXXXXXX
$ aws sts decode-authorization-message --encoded-message $AWS_MESG --query DecodedMessage --output text| jq '.'
...
{
"allowed": false,
"explicitDeny": true,
...
},
"failures": {
"items": []
},
"context": {
"principal": {
"id": "XXXXXXXXXXXXXXXXXX",
"name": "Bamboo_service_account",
"arn": "arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME}"
},
"action": "RunInstances",
...It was discovered that the organization had implemented an AWS SCP that prevented EC2 instances that did not use the IMDSv2 protocol from running. It also ensured that instances that stated with IMDSv2 protocol could not be modified during startup as well. The AWS SCP was similar to the below:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[
{
"Effect":"Deny",
"Action":"ec2:RunInstances",
"Resource":"arn:aws:ec2:*:*:instance/*",
"Condition":{
"StringNotEquals":{
"ec2:MetadataHttpTokens":"required"
}
}
},
{
"Effect":"Deny",
"Action":"ec2:RunInstances",
"Resource":"arn:aws:ec2:*:*:instance/*",
"Condition":{
"NumericGreaterThan":{
"ec2:MetadataHttpPutResponseHopLimit":"3"
}
}
},
{
"Effect":"Deny",
"Action":"*",
"Resource":"*",
"Condition":{
"NumericLessThan":{
"ec2:RoleDelivery":"2.0"
}
}
},
{
"Effect":"Deny",
"Action":"ec2:ModifyInstanceMetadataOptions",
"Resource":"*"
}
]Cause
Currently, Bamboo does not support AWS IMDSv2. There is an ongoing feature request titled "Add support for IMDSv2 on Elastic Agents" that is still attracting attention. When trying to start an elastic instance using Bamboo Stock Images, this action will be denied due to the AWS Service Control Policies (AWS SCP) in place. This is because Bamboo Stock Images only support the IMDSv1 protocol, where using HttpTokens is optional.
Additionally, when the EC2 instance utilizes a different Amazon Machine Image (AMI) that supports the IMDSv2 protocol where using HttpTokens is required, efforts to change the instance metadata option from required to optional are also unsuccessful due to the AWS SCP in place which denies ec2:ModifyInstanceMetadataOptions .
Solution
Currently, the workaround is to ask for an exception in the company policy. The AWS Service Control Policy (AWS SCP) shown earlier can be modified to allow the Bamboo AWS user bypass the SCP in place. A sample ARN will look like arn:aws:iam::355420044059:user/bamboo-aws
To Update an SCP policy in AWS console
Sign in to the AWS Organizations console. You must sign in as an IAM user, assume an IAM role, or sign in as the root user (not recommended) in the organization's management account. On the Service control policies page, choose the name of the policy that you want to update. On the policy's detail page, choose Edit policy. Make any or all of the following changes: You can rename the policy by entering a new name in Policy name. You can change the description by entering new text in Policy description. You can edit the policy text by editing the policy in JSON format in the left pane. Alternatively, you can choose a statement in the editor on the right, and also alter its elements by using the controls. When you're finished, choose Save changes.
The modified policy is shown below
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"StringNotEquals": {
"ec2:MetadataHttpTokens": "required"
},
"StringNotLike": {
"aws:PrincipalARN": [
"arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME}"
]
}
}
},
{
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"NumericGreaterThan": {
"ec2:MetadataHttpPutResponseHopLimit": "3"
}
}
},
{
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"NumericLessThan": {
"ec2:RoleDelivery": "2.0"
}
}
},
{
"Effect": "Deny",
"Action": "ec2:ModifyInstanceMetadataOptions",
"Resource": "*",
"Condition": {
"StringNotLike": {
"aws:PrincipalARN": [
"arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME}"
]
}
}
}
]
}This will allow the AWS user bypass the SCP in place and will be able to run an EC2 instance and also modify InstanceMetadataOptions when using an AMI that supports the IMDSv2 protocol where using HttpTokens is required.
Disclaimer Notice
Please ensure that all implications are thoroughly considered before implementing the workaround. It is also important that the organization comprehensively understands the purpose behind this exemption prior to its use.
Was this helpful?