Bamboo guardrails

The content of this page applies to Bamboo 10.2. If you're looking for information about a different version, select it from the menu in the top-right corner.

背景

アトラシアンは、大手顧客のニーズをサポートすることを約束しています。これには、製品のパフォーマンスとスケーラビリティの継続的な改善が含まれます。インスタンス内のデータ量は、パフォーマンスと安定性の問題の要因になる可能性があります。インスタンスが大きくなるにつれて、時間の経過とともにパフォーマンスが低下するリスクも増えます。多くの場合、これは段階的な低下であり、チームに重大な影響を与えるポイントに到達するまで気づかれない可能性があります。

次の表では観察されたパフォーマンスと安定性への影響を説明して、リスクを軽減するために実行できるいくつかのアクションを提案しています。ガードレールは一部の大手顧客の実際の経験に基づいていますが、必ずしもすべての組織の経験を代表するものではありません。

パフォーマンスと安定性に関する重大な問題が発生するリスクを軽減する方法には、次のようなものがあります。

  • アプリの変更。たとえば、パフォーマンスを向上させるために新しいアプリ バージョンにアップグレード、またはユーザーの管理方法を変更します。

  • インフラストラクチャの変更。たとえば、メモリや CPU を増設、またはクラスターやミラーを実行します。

  • フットプリントを削減するためのデータ クリーンアップ アクティビティ。たとえば、アーカイブ、またはモノリス サイトを分割します。

これらはハード制限ではなく、一部の製品インスタンスは既にこれらのしきい値を超えている可能性があることにご注意ください。異なるデータ タイプ間の相互作用やサイトの負荷など、さまざまな要因によって次に示すような潜在的な影響が発生する可能性とその影響の程度が決まります。あらゆるタイプのリスクと同様に、リスクを特定して計画を立てることが不可欠です。そうすれば、これらのアクションに優先順位付けて将来のパフォーマンス問題の可能性を減らすのに役立ちます。

定義

Product Guardrails are data type recommendations designed to help you identify potential risks and aid you in making decisions about next steps in your instance optimization journey.

Bamboo guardrails

次のガードレールは、スケール リスクを特定して軽減し、インスタンスのクリーンアップに関する意思決定を下すのに役立ちます。

Number of environments

CONTENTTYPE

Number of environments

ガードレール

60,000 deploy environments

この数を調べる方法

How to find the number of deployment projects and environments from Bamboo database

リスク

  • Slower performance of UI screens that depend on deployment information. For example, Agent executable environments.
  • Slower performance of REST API endpoints that depend on deployment information. For instance,  /rest/api/latest/agent/.
  • Stability and performance issues in Bamboo due to increased memory footprint. For example: Bamboo application crashes with OutOfMemory errors, utilizes excessive CPU when garbage collecting, and executes expensive database queries.

  • Longer-running background deployment expiry process that may degrade performance in other areas of the application.

  • Increased maintenance cost of the Bamboo application server due to an increase of heap memory to avoid running out of memory, increased database storage and compute to accommodate deployment environment data.
  • Database deadlocking during the expiry process that’s more prevalent on Microsoft SQL Server.

緩和オプション

  • Increase the Java heap size if experiencing crashes due to OutOfMemory errors or performance issues. For example, for 60,000+ deployment environments, we have observed an estimated required heap of 31G.

    • We recommend discussing this issue with Atlassian Support. We can provide insights based on your data, such as garbage collection logging.

  • Ensure you have Global Expiry configured for deployment results for better data hygiene.

    • Re-examine the required retention period for deployment data based on your organization's requirements.

    • Increasing the frequency of the expiry process can help Bamboo keep deployment results to a minimum, which would shorten the deployment expiration time and reduce performance impact on the rest of the application.

    • As the expiry process can be resource heavy and run times can take hours for large instances, it is advisable to plan when it's scheduled to run. For example, if you could schedule it during times of least application usage, you can avoid resource contention and observable performance issues in Bamboo.

  • You may consider whether you require multiple Bamboo instances for separate teams to accommodate excessive deployment environment usage.

Build log sizes / ActiveMQ bottlenecks

CONTENTTYPE

Build logs size (per job)

ガードレール

25Mb or 100000 lines

この数を調べる方法

While viewing a build log, observe the values of the following sentence immediately above the preview panel:

The build generated xxxxxx lines of output.

It is also possible to verify the logs sizes by checking the stored files on the Bamboo server at <bamboo-shared-home>/builds/plan-<number>/download-data/build_logs.

The Disk space hotspots and cleanup best practices article contains a command that may be used to scan the Bamboo Shared folder for large log files, which can be later matched to their respective builds.

リスク

  • Significant delays or lost build triggers, either manually or automatically via commits.
  • Backend and UI instabilities due to excessive memory usage.
  • Unexpected ActiveMQ errors and system crashes.
  • Challenges with Agent communication, including the "connection refused" and "connection reset" errors found in the logs.

緩和オプション

Linked repositories count

CONTENTTYPE

Number of linked repositories

ガードレール

18000

この数を調べる方法

How to obtain the total count of linked repositories present in Bamboo Data Center

リスク

  • Significant delay in converting plan-level repositories to linked repositories.
  • Performance issues when adding a repository at the plan level.
  • Errors when loading repository configurations.
  • Performance issues on the Repository settings page. To view the page, go to settings, then Build resources, and select Repository settings.
  • Long delays in loading the list of linked repositories.

緩和オプション

  • Regularly review and clean up unused or obsolete repositories.
  • Archive or delete repositories that are no longer needed to reduce the load on Bamboo.
  • To reduce the load on the server, perform operations on repositories in batches rather than all at once.

Excessive test cases in the build result

CONTENTTYPE

Excessive test cases in the build result

ガードレール

  • Small instance ( <5 GB Heap ) - 10,000 test cases per job

  • Large instance ( >5 GB Heap ) - 50,000 test cases per job

この数を調べる方法

How to obtain the test case count

リスク

このガードレールを超えて運用した場合は、次の問題が確認されています。

Memory and storage issues
  1. Frequent full Garbage Collection (GC) pauses.
  2. Out of Memory (OOM) error when opening the Tests tab for a build plan due to High heap consumption.
  3. Tables with test case-related data consume a significant amount of space out of the total available storage, leading to storage limit issues.
パフォーマンスの問題
  1. Long time to process the build results after the build completion.

  2. The expiry process is taking a significant amount of time.

  3. The Build result summary page takes a lot of time to load.


Inconsistent state of buildresults due to large time taken to process them
  1. Build XXX had to be canceled. It was marked as queued but was not present in the queue for (at least) the past 720 seconds.

  2. Build XXX had to be canceled. It was marked as in progress in the database but had no assigned Agents.


緩和オプション

Mitigation options available when exceeding the limit of Bamboo test case count as defined under Bamboo Data Center Guard rails

Size of Git repository

CONTENTTYPE

Size of individual Git repositories

ガードレール

2.0 GB per Git repository

この数を調べる方法

How to determine the size of a Git repository

リスク

When operating above this guardrail, we have observed several issues, including but not limited to:

  • Builds that come from plans linked to the large repository experience delays in getting queued on the Build activity page.

    • This delay has the potential to affect all builds, not just those connected to the large repository, especially for users running Bamboo versions before 9.3.0.

  • Builds consume a substantial amount of time during the Source Code Checkout task on the Agent side.

A less common but more severe side effect involves not just a delay but the disruption of all builds connected to the large repository. This situation arises when the Git operations, triggered by Bamboo to maintain the repository cache or build working directory, time out and are left incomplete. This creates a loop where Bamboo continuously tries to update or rebuild the repository cache (on the Bamboo server/agent) or the build working directory (on the Agent) without success. Learn more about Bamboo Agents and their configuration

Delays and interruptions frequently occur due to prolonged Git operations on large repositories. These happen while Bamboo is updating or rebuilding the repository cache (on the Bamboo server/agent) or the build working directory (on the Agent).

緩和オプション

Limit the repository size to 2.0 GB or less

Limit the repository size to 2.0 GB or less to optimize the speed of Git operations on both Bamboo's server and Agent sides. The following articles outline some steps that can assist you in accomplishing this task:

  1. Reduce the size of Bitbucket cloud repository

  2. Compute various size metrics for a Git repository

  3. Reduce the size of GitLab repository 

Store large files outside your repository

Opt for storing large files outside your repository, such as JFrog Artifactory, Sonatype Nexus, or AWS S3. If it's essential to include the files in the repository, consider using Git LFS.

Note that Git isn’t intended to be used as a backup tool. However, many solutions, such as Arq, Carbonite, and CrashPlan are specifically designed for performing backups.

Ensure that you're running Bamboo 9.3.0 or higher

Ensure that you are running Bamboo 9.3.0 or later, as these versions contain crucial improvements to the change detection process.

Upgrading to version 9.3.0 or later will not resolve the issue entirely. Even after the upgrade, you might encounter delays in builds linked to a large repository being queued. The changes were designed to avoid a situation where prolonged Git operations from a single large repository affect all builds in Bamboo. While builds connected to the large repository may still encounter delays, the positive aspect is that not all of your builds will be affected.

Avoid different URLs in multiple repository configurations pointing to the same repository

Avoid setting up multiple Git repository configurations in Bamboo that point to the same remote repository using different URLs.

This is especially important if the remote repository is larger than 2.0 GB. Bamboo considers these configurations as separate repositories for cache management due to the multiple URLs. If plans associated with these configurations are triggered at the same time, this may lead to delays in builds entering the queue, as Bamboo is still updating or rebuilding the cache for the large repository. This delay can impact all builds, not just those linked to the large repository.

The recommended approach is to have one repository cache folder for each remote repository. Therefore, avoid creating multiple repository configurations in Bamboo that point to a single large repository (> 2.0 GB) with different URLs.

Determine if there are multiple Git repository configurations in Bamboo

Move away from the repository polling trigger

Make sure that none of the plans that are connected to the large repository are actively polling for changes. Check the following articles for more information:

The repository should be set up to notify Bamboo of any changes, rather than Bamboo continuously checking for updates. You can find detailed instructions on how to configure this in the Repository triggers the build when changes are committed article. 

Enable repository caching on agents

Turn on the Enable repository caching on agents option in Bamboo within your large repository configuration. This feature guarantees the creation of a repository copy (cache) on the Agent side. Any modifications to the repository will trigger Bamboo to refresh the cache on the Agent side. This will direct the Source code checkout task to retrieve data from the cache instead of the remote repository.

Don't clean your build working directory

Avoid enabling the Force clean build checkbox for the source code checkout task.

Additionally, don’t enable the Clean working directory after each build option that’s available in the job configuration on the Other options. These settings will force the Agent to perform a fresh checkout of the repository whenever a new build is triggered. Although this may lead to higher disk space consumption (due to the build working directory not being cleaned), it can effectively decrease build times.

Configure Git to ignore LFS objects on the Bamboo server side

If you are using Git LFS on a large repository with Bamboo Specs enabled on Bamboo versions before 9.3.0, you can instruct Git to ignore any LFS objects on the Bamboo server side to optimize Git operations.

This issue has been resolved in Bamboo 9.3.0 and later versions. Bamboo no longer fetches LFS files during a Specs scan.

To exclude all LFS objects from fetch operations, you should add the following line to the ~/.gitconfig file located in the home directory of the user running the Bamboo process:

[lfs] fetchexclude = *

This change might result in:

  • build failures on local Agents that depend on Git LFS files. Builds relying on Git LFS running on remote Agents, elastic Agents, and others will not be affected. Note that local Agents have been removed starting from Bamboo 9.6. Check Bamboo 9.6 release notes

  • Bamboo Specs scanning failures if your Specs code is being tracked using Git LFS.


Adjust SSH proxy timeouts

If you experience timeouts while working with a large Git repository, reach out to Atlassian Support for help in identifying the issue and determining the cause of the timeouts.

Since Bamboo 8.0, you can use a new set of system properties (SSH_PROXY_*) that simplify the adjusting and configuration of timeouts on the SSH proxy used by Bamboo.




最終更新日: 2024 年 12 月 19 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.