Bamboo remote agents lose connection because of 'No space left on device' error
問題
Bamboo remote agents lose connection and unable to reconnect.
atlassian-bamboo-agent.log に次のメッセージが出力される。
INFO | jvm 1 | 2012/02/07 23:58:48 | 2012-02-07 23:58:48,514 WARN [Scheduler_Worker-7] [JmsTemplateMessageSender] JmsException when trying to send message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@115ea42. Retrying...
INFO | jvm 1 | 2012/02/07 23:58:48 | org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
...
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: java.lang.Throwable: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
...
INFO | jvm 1 | 2012/02/07 23:58:48 | Caused by: java.io.IOException: No space left on device
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.io.RandomAccessFile.writeBytes(Native Method)
INFO | jvm 1 | 2012/02/07 23:58:48 | at java.io.RandomAccessFile.write(RandomAccessFile.java:482)
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:387)
INFO | jvm 1 | 2012/02/07 23:58:48 | at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216)atlassian-bamboo.log に次のメッセージが出力される。
2020-12-04 16:47:51,239 ERROR [ActiveMQ Transport: tcp:///1.1.1.1:49810@8080] [MessageDatabase] KahaDB failed to store to Journal, command of type: KAHA_ADD_MESSAGE_COMMAND
java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes(Native Method)
at java.io.RandomAccessFile.write(RandomAccessFile.java:525)
at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335)
at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172)
...
2020-12-04 16:47:51,526 ERROR [ActiveMQ Connection Executor: vm://bamboo#0] [BambooServerSimpleMessageListenerContainer] Encountered non-recoverable JMSException
javax.jms.JMSException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1960)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1979)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:114)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:278)
at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:46)
... 6 more
Caused by: org.apache.activemq.broker.BrokerStoppedException: Broker BrokerService[bamboo] is being stopped
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:205)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.a原因
There's no space left on the disk where $BAMBOO_HOME is located. This will cause the Bamboo messaging broker to shutdown with an unrecoverable exception as its unable to journal messages.
ソリューション
Bamboo must be restarted to recover the broker, but first the disk space issue should be addressed.
- Stop Bamboo
- Clean up or add more storage to the Bamboo server where
$BAMBOO_HOMEis located. - Start Bamboo
In versions Bamboo 6.10.5 or greater, you may also change the location of the ActiveMQ Journal with the below system property if you want to store it on a different location to where $BAMBOO_HOME is:
-Dbamboo.jms.store.directory=/path/to/jms-store
最終更新日: 2020 年 12 月 10 日
Powered by Confluence and Scroll Viewport.