Operations in Jira server fail with connection pool errors in logs
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
問題
症状
- The instance can present with symptoms of a performance problem - it appears requests are taking a long period of time to complete.
- Requests fail in the browser.
- JIRA functionality is failing.
atlassian-jira.log に次のメッセージが表示される。
2015-03-04 16:27:41,193 ajp-bio-8009-exec-715 ERROR anonymous 976x372506x28 1i497bl 172.30.128.60 /secure/Dashboard.jspa [NoModule] There was an error getting a DBCP datasource.
java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:59)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:242)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:73)
at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)
at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:146)
at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:101)
at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)
at org.ofbiz.core.entity.jdbc.SQLProcessor.getConnection(SQLProcessor.java:367)
at org.ofbiz.core.entity.GenericDAO.select(GenericDAO.java:584)
at org.ofbiz.core.entity.GenericHelperDAO.findByPrimaryKey(GenericHelperDAO.java:97)
at org.ofbiz.core.entity.GenericDelegator.findByPrimaryKey(GenericDelegator.java:609)
at org.ofbiz.core.entity.GenericDelegator.findByPrimaryKey(GenericDelegator.java:647)
...
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:246)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:243)
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:53)
... 236 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1134)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 240 more
Or the below may also appear:
2016-01-12 11:24:05,094 StreamsCompletionService::thread-28582 ERROR potatobake 683x843721x16 1mrieu1 127.0.0.1 /plugins/servlet/streams [NoModule] There was an error getting a DBCP datasource.
java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:60)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.trackConnection(DBCPConnectionFactory.java:243)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory.getConnection(DBCPConnectionFactory.java:74)
at org.ofbiz.core.entity.ConnectionFactory.tryGenericConnectionSources(ConnectionFactory.java:69)
at org.ofbiz.core.entity.transaction.JNDIFactory.getConnection(JNDIFactory.java:146)
at org.ofbiz.core.entity.TransactionFactory.getConnection(TransactionFactory.java:136)
at org.ofbiz.core.entity.ConnectionFactory.getConnection(ConnectionFactory.java:59)
at com.atlassian.jira.ofbiz.DefaultOfBizConnectionFactory.getConnection(DefaultOfBizConnectionFactory.java:52)
at com.atlassian.activeobjects.jira.JiraTenantAwareDataSourceProvider$OfBizDataSource.getConnection(JiraTenantAwareDataSourceProvider.java:125)
at com.atlassian.activeobjects.internal.DataSourceProviderActiveObjectsFactory$ActiveObjectsDataSource.getConnection(DataSourceProviderActiveObjectsFactory.java:93)
...
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, general error
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:118)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:247)
at org.ofbiz.core.entity.transaction.DBCPConnectionFactory$1.call(DBCPConnectionFactory.java:244)
at org.ofbiz.core.entity.jdbc.interceptors.connection.ConnectionTracker.trackConnection(ConnectionTracker.java:54)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1112)
at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
... 187 more
原因
JIRA uses a database connection pool to manage connections with the database. When an action within JIRA requests data from the database, it will go through a persistence layer such as Active Objects or OfBiz which will then query the database. When it queries the database, it uses up a connection, which is retrieved from the connection pool.
This timeout happens when the connection pool has been exhausted (as all the connections are currently in use) and can be caused by any of the following:
- There isn't enough connections in the total connection pool for the requirements of JIRA.
- SQL is taking longer than normal to process, so connections are not returned within a timely manner.
- Something is holding onto connections and not releasing them back into the pool.
- Bugs in the database driver, for example the MySQL bug as described in [Archived] - JIRA Stops Functioning due to MySQL FailoverConnectionProxy.
- Bugs in JIRA, for example JRA-39677 - Getting issue details... STATUS .
- The MS SQL database Isolation Level is not set correctly as described in JIRA is running out of database connections frequently on MS SQL Server
- Ignore related ClassNotFoundException, this is red herring, see JIRA throws exception java.lang.ClassNotFoundException for DB driver
回避策
- Ensure the recommended
dbconfig.xmlis being used from the appropriate database docs as specified in Connecting JIRA to a Database. Please note this is version-specific functionality so ensure the appropriate documentation is being used. Increasing the number of connections can mitigate this problem, for example increasing
pool-max-sizein thedbconfig.xml, say changing it from 20 to 50. This will not have any side effects as far as increased overhead on resources.Please ensure the database has enough active connections to serve the maximum set in JIRA. To verify this consult with your DBA or the database documentation.
Please note that you will need to restart the Jira node for it to take effect, In a Jira cluster a rolling restart can be performed.
To identify leaking connections,
logAbandonedcan be added todbconfig.xmlinside thejdbc-datasourcetag, for example in PostgreSQL:<jira-database-config> <name>defaultDS</name> <delegator-name>default</delegator-name> <database-type>postgres72</database-type> <schema-name>public</schema-name> <jdbc-datasource> <url>jdbc:postgresql://dbserver:5432/jiradb</url> <driver-class>org.postgresql.Driver</driver-class> <username>jiradbuser</username> <password>password</password> <pool-min-size>20</pool-min-size> <pool-max-size>20</pool-max-size> <pool-max-wait>30000</pool-max-wait> <pool-max-idle>20</pool-max-idle> <pool-remove-abandoned>true</pool-remove-abandoned> <pool-remove-abandoned-timeout>300</pool-remove-abandoned-timeout> <validation-query>select version();</validation-query> <min-evictable-idle-time-millis>60000</min-evictable-idle-time-millis> <time-between-eviction-runs-millis>300000</time-between-eviction-runs-millis> <pool-test-on-borrow>false</pool-test-on-borrow> <pool-test-while-idle>true</pool-test-while-idle> <logAbandoned>true</logAbandoned> </jdbc-datasource> </jira-database-config>This will add additional overhead as stack traces will be thrown in the logs for abandoned connections, which can have a performance impact.
- Identifying SQL that is taking longer to process than normal can be done by Enable SQL query logging in Jira Data Center.
Jira Server or Data Center is slow with dangerous use of multiple connections error in log is also another known problem that can occur as a result of an improperly configured dbconfig.xml.