Bamboo broker ActiveMQ service stopped with error: KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND java.io.IOException: Async Writer Thread Shutdown
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
要約
Bamboo ActiveMQ service stopped causing agents to be unresponsive.
環境
Observed on Bamboo 8.2.5 but might apply to other versions as well.
診断
The following can be seen in the Bamboo logs:
ERROR [ActiveMQ Journal Checkpoint Worker] [MessageDatabase] Checkpoint failed
java.io.IOException: Permission denied
at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
2022-10-25 15:48:23,236 ERROR [ActiveMQ NIO Worker 30] [MessageDatabase] KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND
java.io.IOException: Async Writer Thread Shutdown
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.enqueue(DataFileAppender.java:164)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.storeItem(DataFileAppender.java:129)
at org.apache.activemq.store.kahadb.disk.journal.Journal.write(Journal.java:961)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1152)
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:1134)
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.addMessage(KahaDBStore.java:566)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.addMessage(KahaDBTransactionStore.java:407)
at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.addMessage(KahaDBTransactionStore.java:161)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:888)
at org.apache.activemq.broker.region.Queue.send(Queue.java:756)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:511)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:477)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:295)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.util.TimeStampingBrokerPlugin.send(TimeStampingBrokerPlugin.java:132)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:154)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:580)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:769)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:335)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:200)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:125)
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:301)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:162)
at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:81)
at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:98)
at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
INFO [ActiveMQ Journal Checkpoint Worker] [DefaultIOExceptionHandler] Stopping BrokerService[bamboo] due to exception, java.io.IOException: Permission denied
java.io.IOException: Permission denied
at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
Agents will then become unresponsive because the broker service stopped:
WARN [scheduler_Worker-2] [RemoteAgentManagerImpl] Detected that remote agent 'bamboo-agent1' has been inactive since <date>
WARN [scheduler_Worker-2] [RemoteAgentManagerImpl] Marking remote agent 'bamboo-agent1' as unresponsive原因
Kahadb lock files are not accessible because of permission issues.
ソリューション
Check the permissions for the <Bamboo-Home>/shared/jms-store/bamboo/KahaDB/ directory and confirm if the Bamboo user is able to access this directory.
In case of a cluster configuration with the shared home deployed on an NFS system, please check if the NFS system has the correct permissions and access lists.
最終更新日 2022 年 11 月 16 日
Powered by Confluence and Scroll Viewport.