Elastic Bamboo switches to regular on-demand EC2 instances after the Spot instance fallback timeout has expired
Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.
Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Except Fisheye and Crucible
Summary
This article provides guidance on how to adjust the fallback timeout to a regular EC2 instance when using AWS Spot configuration and recommends an approach to finding a well-balanced setup that does not affect the delivery of build results.
Environment
Bamboo
Elastic Agents
Diagnosis
You could observe the warning below inside the Support Zip:
1
[elastic-pool-2-thread-5] [RemoteEC2InstanceImpl] Spot request sir-kx86nj9h state is cancelled after 61 seconds, falling back to a regular instance.The warning indicates that a request to create a Spot Instance (with request ID sir-kx86nj9h) was cancelled because it was not fulfilled within the expected time frame (in this case, 61 seconds), given that the spot request was not completed, the system tries to provision a regular (on-demand) EC2 instance as a fallback.
Cause
Bamboo operates on a bidding basis for Spot instances, but does not take dynamic costs into account, and is subject to the time it takes for requests to update. If a transition from BIDDING to IDENTIFIED is not confirmed by AWS within the designated fallback period, Bamboo will automatically switch to a standard on-demand instance.
Users can configure Elastic Bamboo to buy a spot instance of a particular type and fall back to a regular instance after a set amount of time if no instances are available.
As described in:
In this case, the fallback time is 1 minute which is 60 seconds and Bamboo falls back to normal EC2 instance selection after 61 seconds.
Solution
To address the issue, follow the steps below:
Navigate to Bamboo Configuration
Select Spot Instances
Adjust the "Fallback to a regular instance after" property value. It is advisable to increment the value in small amounts
Monitor the creation of Bamboo's Elastic Agent after each adjustment
Continue the adjustments until you achieve a configuration that is balanced and does not negatively impact your build results delivery.
An example of the configuration page is shown below:
Was this helpful?