Bamboo Database Corruption(HSQL) - "APPARENT DEADLOCK!!!"
This KB only applies if you're using Embedded HSQL Database.
Bamboo Service suddenly gets shut down, and after starting the service it shows the following in the logs:
2012-04-27 21:04:20,865 FATAL [WrapperSimpleAppMain] [UpgradeLauncher] Failed to initialise Bamboo container java.lang.IllegalStateException: Could not transition server from 'SETUP' to 'STARTING' ..... Caused by: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is net.sf.hibernate.exception.GenericJDBCException: Cannot open connection ..... Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source. ..... 2012-04-27 21:19:35,474 WARN [Timer-0] [ThreadPoolAsynchronousRunner] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ca1666 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! 2012-04-27 21:19:35,474 WARN [Timer-0] [ThreadPoolAsynchronousRunner] com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@ca1666 -- APPARENT DEADLOCK!!! Complete Status: Managed Threads: 3 Active Threads: 0 Active Tasks: Pending Tasks: com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1e9c5a7 Pool thread stack traces: Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,] Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,] Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,]
Cause1: The lck file wasn't cleaned out during shutdown
The embedded HSQL Database places a *.lck file in BAMBOO_HOME/database folder when the database is being used. Its possible that the file wasn't deleted.
Cause2: The embedded HSQL database in BAMBOO_HOME/database/ folder is corrupted.
HSQL database is not recommended for production Bamboo instances as its susceptible for failures. It is mentioned on the Supported platforms page.
Resolution1: Clear out the lck file
Shutdown Bamboo and make sure there are no orphaned java/Bamboo processes running and delete the BAMBOO_HOME/database/*.lck file
Resolution1: Start with a new database
Install a fresh Bamboo instance and migrate to an external database