問題

高負荷状態の期間に Confluence の速度が低下する。

診断

速度低下中に 20 ~ 30 秒間隔で 10 個のスレッド ダンプを取得し、それらを分析します。

次のように、多数の HTTP スレッドが DB コネクションの取得を待機していることを確認できます。

"http-8080-Processor150" daemon prio=1 tid=0x08543368 nid=0x11aa in Object.wait() [0x665a4000..0x665a51b0]
at java.lang.Object.wait(Native Method)
- waiting on <0x83140488> (a com.mchange.v2.resourcepool.BasicResourcePool)
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAcquire(BasicResourcePool.java:968)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:208)
- locked <0x83140488> (a com.mchange.v2.resourcepool.BasicResourcePool)

または

"TP-Processor3" daemon prio=1 tid=0x1fc5bd30 nid=0x279c in Object.wait() [0x1e47d000..0x1e47dda0]
	at java.lang.Object.wait(Native Method)
	at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
	at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
	- locked <0x32cf7538> (a com.mchange.v2.resourcepool.BasicResourcePool)
	at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)

多数の HTTP スレッドが通常どおり実行されていることも確認できます。

"http-8080-Processor47" daemon prio=1 tid=0x098d5d40 nid=0xe2a runnable [0x757fc000..0x757ff1b0]

通常どおり実行されている HTTP スレッドの数を数えます。

原因

The number of connections configured for the database connection pool is too small to handle the number of incoming requests during peak load time.

For example, after doing the Diagnostics/Test, a large majority of HTTP connections are waiting to acquire a connection to the database. As in the above example, HTTP-thread 150 is waiting for a connection to the database. This indicates that Confluence has almost 150 concurrent things it needs to do.

Almost every HTTP thread requires a connection to the database to perform some sort of work. However, if the number of connections to the DB is too small, this resource will become the bottleneck.

In the above example, we can count that there are usually 30 threads running and actually doing work. Thirty is the default number of database connections in the database pool.

Also, if a rogue or greedy thread consumes many resources, it will force the installation into garbage collection mode. When the JRE does this, it takes all application threads off the CPU. The end result being that requests continue to accrue from users however Confluence is not in a position to complete these requests, and eventually burns out all of its available database connections from the pool.

ソリューション

1. Optimize the database connection pool. This defines the maximum number of connections in the database pool. This number should be at least 10 counts higher, or 25% higher (whichever value is greater) than the number of HTTP threads that will be used in peak time (see below for setting HTTP threads.) This is to account for connections used for background jobs on top of active HTTP threads. (Take a Thread Dump Externally during peak load times and count the number of HTTP-threads in use).

   1. データベースへの直接 JDBC 接続をセットアップ済みの場合:
      <confluence-home> ディレクトリで confluence.cfg.xml ファイルを確認します。次の行を見つけ、要件に合わせて更新します。

      <property name="hibernate.c3p0.max_size">60</property>
      

   2. conf/server.xml でデータ ソースを使用している場合:
      Confluence 5.8.x 以降を使用している場合 (Tomcat 8 が同梱): maxTotal パラメータを構成します。また、maxWait 構成オプションの名前が Tomcat 8 では maxWaitMillis に変更されている点にご注意ください。例については「Apache Tomcat で MySQL データソースを構成する」をご参照ください。

      <Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
      ...
      maxTotal="60"
      />

      
      Confluence 5.7.x 以前を使用している場合 (Tomcat 7 以前が同梱): maxActive パラメータを構成します (この属性の名前は Tomcat 8 で maxTotal に変更されています)。

      <Resource name="jdbc/confluence" auth="Container" type="javax.sql.DataSource"
      ...
      maxActive="60"
      />

   3. Suppose the number defined is approximately equal to the number of HTTP-threads that are running when you take a thread dump, and the rest of the threads are waiting on a connection to the database. In that case, this indicates that you have an insufficient number of connections to your database.
2. Limit the number of HTTP threads available to be closer to the number of database connections available. If you are going to have a bottleneck, then the HTTP thread pool is better than the DB connection pool since fewer resources are consumed while waiting for an HTTP connection.
   How to do this depends on how your application server is configured.

   1. For a standalone configuration using the default connector, change (or add if it doesn't exist) the maxThreads attribute of the Connector defined in <conf-install>\conf\server.xml. If this parameter isn't specified, it will default to 200.

          maxThreads="48"

3. データベースのパフォーマンスを評価します。
4. タイムアウトを確立します。詳細については「データベース クエリのタイムアウトを構成する」をご参照ください。

注意

Synchrony

Synchrony process used for Collaborative Editing feature has its own Database Connection Pool and will use up to 15 DB connections. This pool size cannot be modified, but Synchrony is very efficient at recycling it's connections. Keep this pool in mind when configuring the number of connection in your database.

データベースのコネクション数も構成する必要がある場合があります。SQL Server と Oracle は十分に大きい制限値を持つため、これは通常は PostgreSQL と MySQL に関連します。

MySQL

MySQL の既定の最大コネクション数は 151 です。構成されている最大値を次のように確認できます。

mysql -u confluence -h localhost -p -e "SHOW VARIABLES LIKE '%max_connections%';"

制限を増やす方法については MySQL のドキュメントをご確認ください。

• https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_connections

PostgreSQL

PostgreSQL の最大コネクション数は 100 です。構成されている最大値を次のように確認できます。

psql -U postgres -c 'show max_connections;'

制限を増やす方法については PostgreSQL のドキュメントをご確認ください。

• https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html

SQL Server

Sql Server by default allows 32767 concurrent connections but If you need to adjust it see following community discussion:

https://docs.microsoft.com/en-us/answers/questions/116462/sql-server-maximum-number-of-connections-limit-byp.html