概要

Troubleshooting Performance Problems is not a trivial process and can be quite complicated when it comes to identifying problems with JIRA applications. Given the number of different configurations possible for JIRA applications, such as different versions, plugins, network, hardware configurations and other various factors, it increases the difficulty of solving these problems. This Knowledge Base article is designed to assist in making that job easier to do and also provide guidance in identifying problems and also giving information to Support.

Troubleshooting performance problems often means isolating out each possible cause one-by-one, based off one or more symptoms and verifying if that is the root cause (or one of them). We do this with discovery (collecting diagnostic information on the instance), then analyze that information to see if we can identify problems. It often means changing something small, then testing if that change affected the stability of the instance - this can take quite some time depending upon the complexity of the instance (plugins, hardware configuration and so on).

Commonly Used Terms

JIRA Application Hardware Configuration

JIRA Software Configuration

症状

JIRA applications are generally slow to respond and load in the browser.

Network Connectivity

アプリケーション

Reverse-proxy / Load-balancer

Hardware Resources

Browser Performance Problems

At-capacity / Overused Instance or Server

ディスク速度

データベース

Application bug (JIRA, JVM, Kernel / OS)

Plugin (third-party or bundled)

JIRA applications are unresponsive.

Network Connectivity

アプリケーション

Reverse-proxy / Load-balancer

Hardware Resources

Browser Performance Problems

At-capacity / Overused Instance or Server

ディスク速度

データベース

Application bug (JIRA, JVM, Kernel / OS)

Plugin (third-party or bundled)

CPU usage spikes significantly (e.g.: 80-100% usage)

Hardware Resources

JVM Configuration / Insufficient Memory

At-capacity / Overused Instance or Server 

Application bug (JIRA, JVM, Kernel / OS) 

Plugin (third-party or bundled)

JIRA application crash due to OutOfMemoryError

• Jira server crashes with OutofMemory Java heap space error
• JIRA applications crash due to OutOfMemoryError PermGen space error
• GC overhead limit exceeded error crashes Jira server

JVM Configuration / Insufficient Memory 

Application bug (JIRA, JVM, Kernel / OS) 

Plugin (third-party or bundled)

Unable to obtain a connection from the underlying connection pool

Operations fail in Jira server with connection pool errors in logs

データベース

In order to proceed through this step, please step-through each common problem one-by-one, isolating out if it is a cause for the appropriate symptom.

Network Connectivity

Verify with the Sys Admin / hosting provider if there are any problems with the network and/or server. This can also be done by pinging and/or telnetting to the server on the port JIRA applications are running on, for example:

ping 192.168.1.123
telnet 192.168.1.123 8080

Telnet can be used to verify if the Firewall is blocking that port, and tools such as http://just-ping.com/ can be used to diagnose geographic-specific latency problems. For a comprehensive isolation test, connect to the server over the local network to verify if there are any network problems.

アプリケーション

1. Check the application is up (the Tomcat process will be running).

2. Review the logs to see if there are any errors or exceptions and stack traces. Check https://jira.atlassian.com, the JIRA Server Knowledge Base Archive and Google for those Exceptions/errors.
3. Take 3 thread dumps, 10 seconds apart as per our Generating a Thread Dump docs to analyse. This will give a breakdown of exactly what the instance is doing when it's locked up.

The thread dumps can be analysed to see what may be causing the application to lock up. Thread dumps are only useful when they are taken during application failure or slow performance. If they're taken when the JIRA application instance is stable they are not useful as they will show that JIRA applications are stable, rather than experiencing problems. For examples on how to analyse thread dumps, please review the following sites:

• http://architects.dzone.com/articles/how-analyze-java-thread-dumps
• http://www.javacodegeeks.com/2012/03/jvm-how-to-analyze-thread-dump.html

Additionally:

• Enabling profiling in Administration > System > Troubleshooting and Support > Logging and Profiling and then testing can identify any problems.
• Also, profiling the instance with jProfiler, as per our Use jProfiler to analyse Jira application performance KB will give an exact breakdown of the classes and methods that are the slowest to respond.

Reverse-proxy / Load-balancer

If using a reverse-proxy / load-balancer, try bypassing it and access the instance directly, e.g.: http://192.168.1.123:8080/jira to see if there is any difference.

Analyzing the access logs with our Jira HTTP Requests Log Analyzer is an excellent way of diagnosing if the instance is being overloaded. If so, it can increase response times and those logs will also assist in identifying the culprits.

Hardware Resources

1. Verify the JIRA application Requirements are met and check the CPU load of the JIRA application instance to see if it is high, as if the CPU is consumed by another process (or by a JIRA application), it may be slow to respond.
2. If there are multiple applications on the one server, verify that there are enough resources for all applications. To calculate the required resources for a JVM, add the maximum heap size to the permgen and add 128mb for the JVM. For example:

For a server with Confluence and JIRA applications installed, it would require 3328mb just to provide to those two applications, and you would need to add the overhead of the operating system. At minimum, there should be 4gb available on that machine and it would need to be 64-bit. Also, take into account the requirements of other applications on the same server (for example, a database).

There are known problems if the server hosting JIRA applications is running 32-bit and other JVM applications, as in our JIRA crashes when running as a Windows Service KB.

Browser Performance Problems

Try using a different browser to see if there is any difference in the response - IE (especially IE8) will typically be significantly slower than Chrome to render content. For further assistance diagnosing problems, take a look through our Troubleshooting Client-Side Browser Problems KB.

A definitive isolation test for browser problems is to test cross-browser - if the problem persists across browsers it likely points to the server-side performance.

At-capacity / Overused Instance or Server

• There is no hard and fast rule on this as it often depends on the database stats (issues, custom fields, workflows, users, security schemes and so on) and the JIRA application version. Providing a Support ZIP to Atlassian Support is enough for us to provide some rough recommendations on the instance. It may be that an upgrade is required if the instance is too large for the version.
• Collect and review access logs to see if the instance is experiencing high load.
• Verify if an external application integration using SOAP or REST is overloading the instance.
• Upgrading a test server to the latest version and restoring production onto it is a sure-fire way of identifying if performance problems will be fixed when upgrading.

Analyzing the access logs with our Jira HTTP Requests Log Analyzer is an excellent way of diagnosing if the instance is being overloaded. If so, it can increase response times and those logs will also assist in identifying the culprits.

ディスク速度

Poor disk I/O will cause poor performance with searches, gadgets, and issues with 10+ attachments.  Test the disk speed with the utility and instructions from our Testing Disk Access Speed KB and compare your results with the good/bad/average ranges.

Anti-virus "on access" does not get along with JIRA applications and the indexing directory must be excluded from on-access scanning. In some cases (such as certain versions of Symantec) these exclusions are ignored and the problems will continue.  In this case it may be necessary to remove anti-virus from the server and seek another solution.

For Linux installations consider mounting your filesystems with noatime.  This instructs the OS not to set a timestamp each time a file is read.  When applications access Lucene indexes and cache files thousands of times a second the atime writes can add up to a noticeable slowdown... as much as 30% according to one case study.  Before you change your mount options you must be sure no other application is dependant on atime.  A more conservative setting that would still yield benefits is "relatime".  For more info see Gain 30% Linux Disk Performance with noatime, nodiratime, and relatime.

データベース

1. Verify the load on the database and the DB server - our Test database performance for Jira Server KB can be used for this. Additionally, this should be raised with the DBA for the server, as Atlassian Support are not experts on diagnosing and troubleshooting database problems.
2. Verify the validationQuery has been set as per Surviving Connection Closures and the database connection is configured as per the Connecting JIRA applications to a Database documentation.

既知の問題点

• JRA-9979 - Getting issue details... STATUS  - pre JIRA 5.2, the notificationinstance could grow to massive sizes. In these cases, if JIRA accesses these tables, it would drastically slow down the response time between the database and JIRA. Clearing out this table has helped bring performance back.

JVM Configuration / Insufficient Memory

1. Were JIRA applications recently upgraded? If so, verify that the previous JVM configurations were migrated from the older version.
2. If experiencing OOMEs, increase the maximum memory by small amounts (256mb) as per Increasing JIRA application memory to verify if the problem is still occurring.
3. Garbage Collection logs are required to analyze what's happening with the JVM memory that JIRA applications are using. These can be enabled as in our Troubleshoot Jira Server performance with GC logs KB article.
4. Set up the JVM so that it generates a heap dump automagically when an OOME is thrown, as per our Analyze OutofMemory errors in Jira server with Heap Dumps KB.
5. Check the logs for possible CodeCache warnings, addressed in more details in the Full CodeCache causes Jira server to crash or perform slowly article.

Java VisualVM can also be installed and used to bind to the process that is running JIRA applications. This can be used to analyse CPU utilisation by GC, along with the heap usage. The visualGC plugin is recommended to be installed as it will provide a visual breakdown of each memory space.

• Analysing the GC logs and comparing when an instance is performing slowly is an excellent way of isolating out any memory problems. For example, if the instance is slow to perform during a specific time check if there are any Full GCs occurring during that time, as this could cause the application to respond slowly.
• If the CPU is spiking, check if there are consistent full GCs occurring, as garbage collection is CPU intensive. If there is insufficient memory provided to the JVM, it will need to go through more GC cycles, so it will be using more CPU. Ensure that the JVM has enough memory - increasing it in small increments can help isolate if this is a cause.

Application bug (JIRA, JVM, Kernel/OS)

Check jira.atlassian.com, the JIRA Server Knowledge Base Archive and also the DB/OS/Java vendor to see if there are any bugs that present with performance symptoms. Some examples of OS bugs that have crippled JIRA application performance:

• khugepaged eating 100%CPU
• Linux glibc >= 2.10 (RHEL 6) malloc may show excessive virtual memory usage

Plugin (third-party or bundled)

Plugins, especially third-party or in-house (developed internally), can cause a significant number of problems with JIRA applications. They can consume the DBCP, memory (heap usage), slow down issue operations and cause any number of other problems. Best practice is to verify if the performance problems are still experienced when no plugins are enabled. This can be done by enabling Safe Mode in the UPM will verify whether or not a plugin is a problem, however it comes at the detrimental impact of reducing the instance's functionality. This can be a hard case to make to the business, so we recommend this be done on a replicated test instance (that is, one identical to production). This can also be done by isolation testing as below:

1. Enable Safe Mode and test to see if the problem is still reproducible. If it is not, it implies a plugin is causing a slow-down. If the problem is no longer reproducible, continue with the below steps, otherwise this rules out plugins altogether.
2. Enable one single plugin and test.
3. Repeat step 2 until the problem replicates itself - the most recently enabled plugin can be considered the culprit.
4. Disable all other plugins except for the one identified in step 3 and then verify if it is indeed the problem (rather than that plugin being used in combination with any others). If so, raise it with the plugin developers as from the Atlassian Marketplace or with Atlassian if it's one of our plugins.

Additionally, verify that all your plugins are valid and up-to-date for the JIRA application version. It is highly common that an incompatible plugin will cause problems with a JIRA application instance.

サポートに情報を提供する

Often when dealing with performance problems, Atlassian Support will request further info on the instance so we can troubleshoot the performance problems. In order to do so, please provide the following information:

1. What are the specs of the server running JIRA applications (CPU/RAM/OS & Architecture/Physical or Virtual)?
2. What other applications are running on that server?
3. Which other applications use the database server?
4. Which browser and version does the error happen in?
5. Does it happen in any other browsers?
6. If the instance is performing slowly, which pages are performing slowly (provide some sample URLs).
7. When was the instance performing slowly (so we can check the logs)?
8. Provide GC logs, as in JVM Configuration / Insufficient Memory.
9. Provide Access logs if using a Reverse-proxy, as in Reverse-proxy / Load-balancer.
10. The results of Disk Speed tests, as from that section.
11. Generate a Support ZIP and provide it along with the above information.