All Remote Agent disconnecting or Transport Connection errors
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
要約
Intermittent All Remote Agents drop from Bamboo. Or intermittent Transport Connection Channel was inactive for too (>300000) long errors and evidence of a stuck ActiveMQ VMTransport thread.
診断
All Remote Agent intermittently drop from Bamboo and Intermittent errors in logs like the following:
WARN [ActiveMQ InactivityMonitor Worker] [Transport] Transport Connection to: tcp://10.10.10.10:49705 failed: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too (>300000) long: tcp://10.10.10.10:49705The $BAMBOO_HOME/logs/emergency-bamboo.log thread dumps will contain instances where one of the VMTransport threads are stuck in a send:
"ActiveMQ VMTransport: vm://bamboo#1-623" daemon prio=5 tid=0x00000000009ca35f nid=0x9ca35f timed_waiting [496289df]
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:192)
at org.apache.activemq.transport.nio.NIOOutputStream.flush(NIOOutputStream.java:132)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194)
at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335)
at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:94)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:116)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1479)
at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:977)
at org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:933)
at org.apache.activemq.broker.TransportConnection.dispatchAsync(TransportConnection.java:943)
at org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:713)
at org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:121)
at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:48)
at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:775)
at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:556)
at org.apache.activemq.broker.region.Topic.send(Topic.java:484)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:508)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:459)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.util.TimeStampingBrokerPlugin.send(TimeStampingBrokerPlugin.java:131)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:578)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:336)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:275)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)原因
When Bamboo and a Remote Agent have a network issue, Bamboo checks every 5 minutes if the TCP session is still active. Bamboo does not timeout the session for write operations. Eventually, the session does timeout depending on other settings. There are ways that Bamboo can restart the time even with no action.
ソリューション
From Bamboo 9.4.0, new installations will default to the Agent timeout settings proposed on this KB article. Upgraded Bamboo instances will still carry the previous settings unchanged. We highly recommend you update them following the instructions below. Please check BAM-20858 - Getting issue details... STATUS for more information.
Adding transport.soWriteTimeout under the Bamboo UI > Cog > Overview > System> General configuration > Broker URL, will allow Bamboo to shut down the sessions if the TCP session has timed out.
- We now check dead transports every 90 seconds by reducing the
maxInactivityDuration - If something gets stuck in a socket write, now it will be timed out in 45 seconds by adding
transport.soWriteTimeout
What we are recommending:
- Edit the
$BAMBOO_HOME/bamboo.cfg.xml Update the
bamboo.jms.broker.uriand include the changes tomaxInactivityDurationand transport.soWriteTimeout. Keep the specified protocol (nio,tcp,ssl, etc) unchanged.<property name="bamboo.jms.broker.uri">nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=90000&transport.soWriteTimeout=45000</property>- Restart Bamboo for the changes to take affect
Please note, if you are making these changes directly in the UI configuration and not within the XML, there is no need to escape the ampersand with & Bamboo will still need to be restarted
Still having problems?
アトラシアン サポートにお問い合わせください。喜んでお手伝いします。可能な限り詳細な情報をご提供ください。
- 影響を受けているアプリケーションのログ (該当する場合)
- エラー メッセージのスクリーンショット (ログに含まれていない場合)
- 過去に行った手順についての情報