Bamboo Elastic Agents fail to start up or get stuck in a pending state.
プラットフォームについて: Data Center - この記事は、Data Center プラットフォームのアトラシアン製品に適用されます。
このナレッジベース記事は製品の Data Center バージョン用に作成されています。Data Center 固有ではない機能の Data Center ナレッジベースは、製品のサーバー バージョンでも動作する可能性はありますが、テストは行われていません。サーバー*製品のサポートは 2024 年 2 月 15 日に終了しました。サーバー製品を利用している場合は、アトラシアンのサーバー製品のサポート終了のお知らせページにて移行オプションをご確認ください。
*Fisheye および Crucible は除く
本記事で説明している手順は、現時点でのものとなります。そのため、一部のお客様で特定の状況下で動作したという報告がありますが、正式にサポートされているわけではなく、お客様の特定のシナリオで動作することを保証するものではありません。
本番環境での実施の前に一通り非本番環境で検証し、成功しなかった場合にはサポートされている代替案にフォール バックしてください。
要約
Bamboo Elastic Agents get stuck in a pending state when triggered or started manually. Additionally, there are scenarios where the EC2 instance fails to reach a running state, displaying the error message, "Elastic Bamboo Error: EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXXXXXXXX failed."
環境
This solution has been tested in the following environments, 9.2.7, 9.2.14, 9.3.4 and 9.4.3 but the solution will be applicable for other Bamboo versions as well.
診断
As mentioned earlier, there are two alternative scenarios:
シナリオ 1
Bamboo elastic agents get stuck in a pending state for a long time. View the instance using ,select 
> Elastic Bamboo > Instances. This shows the below as the state of the instance:
Scenario 2
The EC2 instance fails to start with the error message below which can be seen in the <BAMBOO-HOME>/logs/atlassian-bamboo.log
Elastic Bamboo Error : EC2 Instance failed to start: EC2 instance order for image ami-XXXXXXXXXXXXXX failed.
(com.amazonaws.services.ec2.model.AmazonEC2Exception : You are not authorized to perform this operation. User: arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME} is not authorized to perform: ec2:RunInstances on resource: arn:aws:ec2:{region}:{ACCOUNT_ID}:{RESOURCE_TYPE}/instance/* with an explicit deny in a service control policy. Encoded authorization failure message: XXXXXXXXXXXXX
....After attempting to decode the Encoded authorization failure as shown below, we can see a failure due to an AWS Service Control Policy.
$ export AWS_MESG=XXXXXXXX
$ aws sts decode-authorization-message --encoded-message $AWS_MESG --query DecodedMessage --output text| jq '.'
...
{
"allowed": false,
"explicitDeny": true,
...
},
"failures": {
"items": []
},
"context": {
"principal": {
"id": "XXXXXXXXXXXXXXXXXX",
"name": "Bamboo_service_account",
"arn": "arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME}"
},
"action": "RunInstances",
...It was discovered that the organization had implemented an AWS SCP that prevented EC2 instances that did not use the IMDSv2 protocol from running. It also ensured that instances that stated with IMDSv2 protocol could not be modified during startup as well. The AWS SCP was similar to the below:
[
{
"Effect":"Deny",
"Action":"ec2:RunInstances",
"Resource":"arn:aws:ec2:*:*:instance/*",
"Condition":{
"StringNotEquals":{
"ec2:MetadataHttpTokens":"required"
}
}
},
{
"Effect":"Deny",
"Action":"ec2:RunInstances",
"Resource":"arn:aws:ec2:*:*:instance/*",
"Condition":{
"NumericGreaterThan":{
"ec2:MetadataHttpPutResponseHopLimit":"3"
}
}
},
{
"Effect":"Deny",
"Action":"*",
"Resource":"*",
"Condition":{
"NumericLessThan":{
"ec2:RoleDelivery":"2.0"
}
}
},
{
"Effect":"Deny",
"Action":"ec2:ModifyInstanceMetadataOptions",
"Resource":"*"
}
]原因
Currently, Bamboo does not support AWS IMDSv2. There is an ongoing feature request titled "Add support for IMDSv2 on Elastic Agents" that is still attracting attention. When trying to start an elastic instance using Bamboo Stock Images, this action will be denied due to the AWS Service Control Policies (AWS SCP) in place. This is because Bamboo Stock Images only support the IMDSv1 protocol, where using HttpTokens is optional.
Additionally, when the EC2 instance utilizes a different Amazon Machine Image (AMI) that supports the IMDSv2 protocol where using HttpTokens is required, efforts to change the instance metadata option from required to optional are also unsuccessful due to the AWS SCP in place which denies ec2:ModifyInstanceMetadataOptions .
ソリューション
Currently, the workaround is to ask for an exception in the company policy. The AWS Service Control Policy (AWS SCP) shown earlier can be modified to allow the Bamboo AWS user bypass the SCP in place. A sample ARN will look like arn:aws:iam::355420044059:user/bamboo-aws
To Update an SCP policy in AWS console
Sign in to the AWS Organizations console. You must sign in as an IAM user, assume an IAM role, or sign in as the root user (not recommended) in the organization's management account. On the Service control policies page, choose the name of the policy that you want to update. On the policy's detail page, choose Edit policy. Make any or all of the following changes: You can rename the policy by entering a new name in Policy name. You can change the description by entering new text in Policy description. You can edit the policy text by editing the policy in JSON format in the left pane. Alternatively, you can choose a statement in the editor on the right, and also alter its elements by using the controls. When you're finished, choose Save changes.
The modified policy is shown below
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"StringNotEquals": {
"ec2:MetadataHttpTokens": "required"
},
"StringNotLike": {
"aws:PrincipalARN": [
"arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME}"
]
}
}
},
{
"Effect": "Deny",
"Action": "ec2:RunInstances",
"Resource": "arn:aws:ec2:*:*:instance/*",
"Condition": {
"NumericGreaterThan": {
"ec2:MetadataHttpPutResponseHopLimit": "3"
}
}
},
{
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"NumericLessThan": {
"ec2:RoleDelivery": "2.0"
}
}
},
{
"Effect": "Deny",
"Action": "ec2:ModifyInstanceMetadataOptions",
"Resource": "*",
"Condition": {
"StringNotLike": {
"aws:PrincipalARN": [
"arn:aws:iam::{ACCOUNT_ID}:{RESOURCE_TYPE}/{RESOURCE_NAME}"
]
}
}
}
]
}This will allow the AWS user bypass the SCP in place and will be able to run an EC2 instance and also modify InstanceMetadataOptions when using an AMI that supports the IMDSv2 protocol where using HttpTokens is required.
Disclaimer Notice
Please ensure that all implications are thoroughly considered before implementing the workaround. It is also important that the organization comprehensively understands the purpose behind this exemption prior to its use.