Apache and Tomcat load balancing

概要

The following is a description of how to set up a Confluence Cluster on a Windows machine using Apache and mod_jk to handle the loadbalancing.

The characteristics of this cluster are:

• Session affinity: sessions are associated with single servers.
• Failover: if a server dies, a connection will be directed to the nearest available server. (NOTE: sessions are not replicated)
• Failback: when a server comes back online, it will rejoin the cluster.
• Weighted load balancing: the load balancing can be controlled to take into account machine differences. (See the mod_jk documentation for details on this.)

What do you need?

1. Download and install one copy of Apache httpd. Do not install Apache as a service, but set it to listen on port 8080. (Tested with Apache httpd 2.0.55.)
2. Download the latest version of mod_jk. Copy this file into the Apache modules/ directory and rename it to mod_jk.so. (Tested with JK-1.2.19.)
3. Download and extract one copy of the ZIP distribution of Apache Tomcat. (Tested with Tomcat 5.5.)
4. Download JDBC drivers for the external database you will be using. Put the drivers in Tomcat's common/lib/ directory. (Tested with Postgresql 8.1, postgresql-8.1-404.jdbc3.jar).

Apache 設定

Edit the main Apache config file, conf/http.conf:

• add the following immediately after the other LoadModule directives:

  LoadModule jk_module modules/mod_jk.so
  

• add the following just before the end of the file:

  JkWorkersFile conf/workers.properties
  JkLogFile logs/mod_jk.log
  JkLogLevel info
  
  JkMount /confluence loadbalancer
  JkMount /confluence/* loadbalancer
  

Create a workers.properties file in the Apache conf/ directory. This version of the workers.properties file is configured to use 2 Tomcat instances: tomcat1 and tomcat2.

worker.list=loadbalancer

worker.tomcat1.port=18081
worker.tomcat1.host=localhost
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=1
  
worker.tomcat2.port=28081
worker.tomcat2.host=localhost
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=1

worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcat1, tomcat2
worker.loadbalancer.method=Busyness

Tomcat 環境設定ファイル

The Tomcat configuration below will run multiple instances from the same binaries in the main Tomcat directory. For complete documentation of this configuration, see the RUNNING.txt file in the Tomcat distribution.

Create instance home directories

Create a directory for each instance of Tomcat, somewhere outside where you installed Tomcat. For example, if you extracted Tomcat to /opt/apache/tomcat-5.5, your instances could be in /var/tomcat-instances/tomcat1, /var/tomcat-instances/tomcat2. These folders will be referred to as the instance home directories.

Copy the following folders from the Tomcat installation directory into each instance home directory. Some of the folders may be empty, but copy them anyway.

• conf
• logs
• shared
• webapps

Configure server.xml in each instance

Edit conf/server.xml in the instance home directories to include the Confluence application and have distinct listen ports for Server, HTTP Connector and AJP13 Connector. All nodes can use the same Confluence webapp as long as you set confluence.home via a system property (see startup scripts below).

Attached are two sample configurations:

• tomcat1/conf/server.xml - listens on port 18080 (http) and 18081 (ajp13)
• tomcat2/conf/server.xml - listens on port 28080 (http) and 28081 (ajp13)

To use these sample config files, you will need to edit them to set the Confluence web-app location and the data source configuration.

If editing the configuration files yourself, the points to note are:

• 'Server' port must be distinct
• 'Connector' for HTTP must be uncommented and use a distinct port. Use this port for testing the node individually.
• 'Connector' for AJP13 must be uncommented and use a distinct port. This port must match the port of the worker in the Apache workers.properties.
• 'Engine' for localhost must have jvmRoute matching the name of the worker in Apache's workers.properties.
• 'Context' for Confluence must be added inside the 'Host' tag, and include a 'Resource' for the datasource, as per normal Confluence installation under Tomcat.

Create a startup script for each instance

The startup scripts for each instance must set the CATALINA_BASE environment variable and confluence.home system property. The variables in the sample scripts below should reference:

• CATALINA_HOME - Tomcat installation directorty
• CATALINA_BASE - Tomcat instance home directory (distinct for each node)
• JRE_HOME - Java runtime directory
• JAVA_OPTS - include a confluence-home system property (distinct for each node)

tomcat1/startup.bat:

set CATALINA_HOME=C:\home\mryall\opt\apache\apache-tomcat-5.5.16
set CATALINA_BASE=C:\home\mryall\var\tomcat-instances\tomcat1
set JRE_HOME=C:\Java\jre1.5.0_06
set JAVA_OPTS=-Dconfluence.home=C:\home\mryall\data\confluence\cluster\tomcat1 -Xmx512m
%CATALINA_HOME%\bin\startup.bat

tomcat2/startup.bat:

set CATALINA_HOME=C:\home\mryall\opt\apache\apache-tomcat-5.5.16
set CATALINA_BASE=C:\home\mryall\var\tomcat-instances\tomcat2
set JRE_HOME=C:\Java\jre1.5.0_06
set JAVA_OPTS=-Dconfluence.home=C:\home\mryall\data\confluence\cluster\tomcat2 -Xmx512m
%CATALINA_HOME%\bin\startup.bat

Continue setting up Confluence

Follow the Confluence Cluster Installation procedure with the steps following the app server setup.

トラブルシューティング

General advice

The above tomcat configurations enable HTTP connectors on each Tomcat instance so that you can connect to the nodes individually. To check whether the load balancer (Apache & mod_jk) is causing the problem, try connecting to the individual Tomcat instances. Please note that you should not allow users to directly access individual nodes in production mode: You don't want people to bookmark nodes since the node details might change, or single nodes may be taken out of the cluster for maintenance while the cluster itself is still available.

Session-affinity doesn't seem to be working?

Ensure the name you use for your worker in workers.properties (e.g. tomcat1) matches the jvmRoute attribute of the engine tag in your Tomcat server.xml. For an example, search for 'Engine' in the attached sample config.

For troubleshooting your Confluence cluster, see Cluster Troubleshooting.

参考

一般

http://raibledesigns.com/tomcat/ http://httpd.apache.org/

Tomcat Clustering support

http://tomcat.apache.org/tomcat-5.0-doc/cluster-howto.html http://tomcat.apache.org/tomcat-5.0-doc/balancer-howto.html http://tomcat.apache.org/tomcat-3.3-doc/mod_jk-howto.html
Clustering and Load Balancing in Tomcat 5, Part 1
Clustering and Load Balancing in Tomcat 5, Part 2