Enabling JMX counters for performance monitoring
JMX を使用して何を監視できますか?
It is possible to monitor various statistics using JMX counters within Bitbucket Server. Below are some examples of some statistics that can be monitored.
Thread pools
| Thread pool | 説明 | Object name |
|---|---|---|
| IoPumpThreadPool | Threads that handle external process IO | com.atlassian.bitbucket.thread-pools:name=IoPumpThreadPool |
| ScheduledThreadPool | Thread pool that takes care of several miscellaneous scheduled tasks | com.atlassian.bitbucket.thread-pools:name=ScheduledThreadPool |
| EventThreadPool | Threads that dispatch events to @EventListenermethods | com.atlassian.bitbucket.thread-pools:name=EventThreadPool |
Thread pool attributes
名前 | 説明 |
|---|---|
| ActiveCount | Returns the approximate number of threads that are actively executing tasks. |
| MaximumPoolSize | Returns the maximum allowed number of threads. |
| PoolSize | Returns the current number of threads in the pool. |
| QueueLength | The number of tasks awaiting execution by the thread pool. |
| LargestPoolSize | The largest number of threads that have ever been simultaneously in the pool. |
| CompletedTaskCount | The approximate total number of tasks that have completed execution. Because the states of tasks and threads may change dynamically during computation, the returned value is only an approximation, but one that does not ever decrease across successive calls. |
Repositories (com.atlassian.bitbucket:name=Repositories)
| 名前 | 説明 |
|---|---|
| count | Number of repositories currently configured in Bitbucket server across all projects |
Scm Statistics (com.atlassian.bitbucket:name=ScmStatistics)
| 名前 | 説明 |
|---|---|
| Pulls | Number of scm pulls serviced by this instance since it was started |
| Pushes | Number of scm pushes received by this instance is it was started |
Ticket statistics
Bitbucket server uses 'tickets' as a mechanism for creating back-pressure to prevent the system from being overloaded with requests. There are two types of tickets used by Bitbucket server, hosting tickets and command tickets.
Hosting tickets (com.atlassian.bitbucket:name=HostingTickets): Limits the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently.
Command tickets(com.atlassian.bitbucket:name=CommandTickets): Limits the number of SCM commands, such as: `git diff`, `git blame`, or `git rev-list`, which may be running concurrently.
Bitbucket server supports the following metrics for each ticket type.
| 名前 | 説明 |
|---|---|
| Available | The number of tickets available for acquisition (lower number means higher load) |
| LastRejection | The timestamp of the last rejected ticket, or null if no tickets have been rejected |
| 名前 | The name of the ticket bucket either 'scm-command' or 'scm-hosting' |
| OldestQueuedRequest | The timestamp at which the oldest queued request started waiting, or null if there are no queued requests |
| QueuedRequests | The number of requests currently waiting for an available ticket |
| 合計 | The maximum number of tickets that can be acquired concurrently before back-pressure is applied |
| Used | The number of tickets that have been acquired (higher number means higher load) |
Event Statistics (com.atlassian.bitbucket:name=EventStatistics)
| 名前 | 説明 |
|---|---|
Dispatched Count | Total number of listener callbacks that have been performed. An event that is delivered to 10 listeners counts as 10 dispatches |
| Last Rejection | Date of the last event being rejected, or null if no event has been rejected |
| Published Count | Total number of events delivered. An event that is delivered to 10 listeners counts a 1 event. |
| Queue Capacity | Maximum number of event callbacks that can be queued before events are rejected |
| Queue Length | Number of event callbacks that have been queued but haven't been dispatched yet. |
| Rejected Count | Total number of events that were not dispatched because the event queue was full |
| Remaining Queue Capacity | Remaining number of event callbacks that can be queued before events are rejected |
Cluster Lock Statistics (com.atlassian.bitbucket:name=ClusterLocks)
| 名前 | 説明 |
|---|---|
Locked Count | Number of cluster locks that are currently held by this node |
Queued Thread Count | Number of threads on this node that are currently blocked waiting for a lock |
Total Acquired Count | Total number of times a cluster lock was acquired on this node since startup |
Total Acquire Error Count | Number of times an exception was thrown while trying to acquire a cluster lock on this node since startup |
Total Acquire Time Millis | Total time in milliseconds that any thread on this node has spent acquiring a lock (including time blocked waiting for a lock to become available) |
Total Released Count | Total number of times a cluster lock was released on this node since startup |
Total Release Error Count | Total number of times an exception was thrown while releasing a cluster lock on this node since startup |
SSH Session Statistics (com.atlassian.bitbucket:name=SshSessions)
| 名前 | 説明 |
|---|---|
Active Session Count | Number of currently active SSH session |
Max Active Session Count | Highest number of concurrently active SSH sessions since the last startup |
| Session Closed Count | Total number of SSH sessions that have been closed since the last startup |
Session Created Count | Total number of SSH sessions that have been created since the last startup |
Session Exception Count | Total number of SSH sessions that have been terminated because an exception was thrown from the SSH command run. |
Interesting 3rd party library attributes
Bitbucket Server exposes the JMX attributes from number of third party libraries. Listed below is a sample of the attributes that are particularly interesting from an operations perspective.
HikariCP - (com.zaxxer.hikari:type=Pool (bitbucket))
名前 | 説明 |
|---|---|
ActiveConnections | Active Connections (in use) |
IdleConnections | Idle Connection count |
ThreadsAwaitingConnection | The number of threads waiting for a connection (when all available connections are in use) |
| TotalConnections | Total Connections |
Hibernate - (org.hibernate.core:sessionFactory=bitbucket.core,serviceRole=org.hibernate.stat.Statistics,serviceType=org.hibernate.stat.internal.ConcurrentStatisticsImpl)
| 名前 | 説明 |
QueryCacheHitCount | Global number of cached queries successfully retrieved from cache |
QueryCacheMissCount | Global number of cached queries not found in cache |
SecondLevelCacheHitCount | Global number of cacheable entities/collections successfully retrieved from the cache |
SecondLevelCacheMissCount | Global number of cacheable entities/collections not found in the cache and loaded from the database |
Expose JMX MBeans within Bitbucket Server
To enable Bitbucket Server to publish specific statistics using JMX you need to
- Modify the
bitbucket.propertiesfile. - Create a JMX password file for secure access to JMX monitoring.
- Modify the
setenv.shfile to enable Bitbucket Server to expose JMX Mbeans.
These changes will not take effect until Bitbucket Server has been restarted.
Modify the bitbucket.properties file
To modify (or create) the bitbucket.properties file
Create the
bitbucket.propertiesfile, in thesharedfolder of your Bitbucket home directory. Take care to use the standard format for Java properties files.Thebitbucket.propertiesfile is created automatically if you previously performed a database migration.Add this property to the file.
jmx.enabled=true
Set up the JMX password file
To set up a JMX password file to secure access to JMX monitoring
Create a file named
jmx.access.This file will contain password information. Ensure the file is only readable by the secure user Bitbucket Server will run under. However, note that if the Bitbucket Server user cannot read the file Bitbucket Server will fail to start.
Edit the
jmx.accessfile to include this property and save the file.monitorRole=<password>If you wish to use a username other than
monitorRoleorcontrolRoleyou will need to modify the jmxremote.access file located in the /lib/management/directory of the installed Java.
Modify the Bitbucket Server environment file
To modify the setenv.sh (for Windows setenv.bat) files to enable JMX monitoring for Bitbucket Server
Within the
bindirectory, locate the filesetenv.sh(for Windowssetenv.bat) and change these properties.JMX_REMOTE_AUTH=password JMX_REMOTE_PORT=3333 RMI_SERVER_HOSTNAME=-Djava.rmi.server.hostname=<hostname> JMX_PASSWORD_FILE=<path>/jmx.access- Restart Bitbucket Server.
Expose JMX MBeans when Bitbucket Server is run as a Windows service
To expose JMX MBeans when Bitbucket Server is run as a Windows service.
- Stop the Bitbucket Server service.
Open the command line prompt and enter.
cmdNavigate to the Bitbucket Server
bindirectory.cd <Bitbucket Server Installation dir>\binRun this command.
tomcat8w //ES//AtlassianBitbucket ServerIn the window that appears, click on the Java tab to see the list of current startup options. Under "Java Options:" form, input the value
-Dcom.sun.management.jmxremote.port=<JMX_REMOTE_PORT> -Djava.rmi.server.hostname=<hostname> -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=<JMX_PASSWORD_FILE>Ensure the owner of this password file is the secure user Bitbucket Server will run as. If the Bitbucket Server user cannot read the file, Bitbucket Server will fail to start.
Replace the values within the < > characters.
JMX_REMOTE_PORT=3333 JMX_PASSWORD_FILE=<path>\jmx.accessRestart Bitbucket Server Service.
Verify JMX is configured correctly
These steps use JConsole to test that JMX has been configured correctly. JConsole is a utility that ships with the Oracle JDK.
To start the jconsole utility, from a command line prompt enter
jconsole
Create a new JConsole connection with similar connection settings.
bitbucket the hostname of the instance of Bitbucket Server to monitor 3333
the JMX port number previously configured.
username, password values configured within the JMX password file jmx.access.Click Connect.
When configured correctly, you will see these properties.
| com.atlassian.bitbucket |
|
| com.atlassian.bitbucket.thread-pools |
|
Example performance dashboard
This dashboard was generated using Java Mission Control that ships with the Oracle JDK (since 1.7u40). See the documentation that comes with your JMX client of choice for more information.
Configuring JMX to use SSL
You can find information about the options for configuring JMX to use SSL in the setenv files. Comprehensive documentation is available from Oracle.