Bamboo startup is taking a long time or failing at the "Reticulating splines" log entry due to Crowd Sync
問題
Bamboo startup hangs or takes a long time at the "Reticulating splines" stage in the logs:
2016-10-11 16:42:54,621 INFO [localhost-startStop-1] [BambooContainer] Reticulating splines...Bamboo may still continue to log messages about schema updates even after this line:
2020-01-24 03:51:48,232 INFO [localhost-startStop-1] [BambooContainer] Reticulating splines...
2020-01-24 03:51:48,303 INFO [active-objects-init-compatibility-tenant-0] [ActiveObjectUpgradeManagerImpl] Starting upgrade of data model, current version is 0
2020-01-24 03:51:48,303 INFO [active-objects-init-compatibility-tenant-0] [ActiveObjectUpgradeManagerImpl] Finished upgrading, model is up to date at version 0Bamboo may eventually error and fail to start with a log line similar to the below:
2016-10-11 22:27:58,050 FATAL [localhost-startStop-1] [BambooContainer] Cannot start bamboo
java.lang.IllegalStateException: Could not transition server from 'SETUP' to 'STARTING'
...
Caused by: javax.xml.bind.DataBindingException: javax.xml.bind.UnmarshalException
- with linked exception:
[java.io.IOException: chunked stream ended unexpectedly]
at javax.xml.bind.JAXB.unmarshal(JAXB.java:226)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:349)
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getMemberships(RestCrowdClient.java:834)
at com.atlassian.crowd.integration.atlassianuser.UserGroupFetcher.fetch(UserGroupFetcher.java:45)診断
環境
- Bamboo <= 6.5.x
- Bamboo is integrated with Crowd for User Management.
Diagnostic Steps
Run the below cURL to the Crowd Server's group membership endpoint to understand how long it takes to run outside of Bamboo. If the time of this call corresponds with how long it takes Bamboo to startup it's a very good indicator this is the root cause of the slow startup.
time curl -u <bamboo_application_name>:<bamboo_application_password> -X GET http://<crowd_url>:<crowd_port>/rest/usermanagement/1/group/membershipThread dumps of the Bamboo Java process will show that the localhost-startStop-1 thread is stuck in a stack trace similar to the below:
"localhost-startStop-1" #13 daemon prio=5 os_prio=0 tid=0x00007f4ea8002000 nid=0x1478e runnable [0x00007f4eed8e3000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
...
at javax.xml.bind.JAXB.unmarshal(JAXB.java:223)
at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:349)
at com.atlassian.crowd.integration.rest.service.RestCrowdClient.getMemberships(RestCrowdClient.java:834)原因
Bamboo is attempting to retrieve a group membership data from a Crowd REST endpoint (/rest/usermanagement/1/group/membership) and this call is taking a long time to return, either due to the Crowd Server's performance or the sheer size of the dataset.
If the call on startup is eventually erroring with javax.xml.bind.DataBindingException: javax.xml.bind.UnmarshalException, this suggests that a network infrastructure element such as a proxy between Bamboo and Crowd and is actually terminating the request prematurely due to the duration or size of the request. If Bamboo encounters this error on startup, it will fail to startup due to this bug:
- BAM-15353 - Getting issue details... STATUS
回避策
The REST endpoint displays every group and user membership of that group from all the directories available to the Bamboo application in Crowd. Disabling Allow all to authenticate and manually adding groups in the Application configuration will not minimize the data set.
Ensure you are running at least Crowd 2.10.1 to ensure you have the following fix to the performance of the dependent REST endpoint (latest version is advised to receive all performance fixes): CWD-4344 - Getting issue details... STATUS
Minimize the size of the dataset being returned from Crowd to Bamboo: Create a separate directory with a refined user and group filter specifically for the Bamboo application in Crowd rather than using the entire organizational directory.
If experiencing a javax.xml.bind.DataBindingException: javax.xml.bind.UnmarshalException:
- Configure Bamboo to connect directly to Crowd and bypass any proxy servers. The Crowd URL used by Bamboo can be modified in the UI at Bamboo Administration >> User Repositories or manually on the filesystem at
<bamboo-home>/configuration/xml-data/crowd.propertieswith the propertycrowd.base.url - Or tweak the configuration for the reverse proxy in-front of Crowd to increase the request timeout and request size limits to be high enough to let the request complete, despite its large size.
ソリューション
Upgrade to Bamboo 6.6+ which features Embedded Crowd with more reliable and efficient functionality for synchronizing with external Crowd Servers:
- BAM-11376 - Getting issue details... STATUS