Generate a Thread Dump Externally

If Stash stops responding or is showing poor performance, providing thread dumps to Support can help diagnose the problem.

Generating a Thread Dump for Windows

Method 1 - Windows scripts

We now have scripts for generating thread dumps externally on Windows. Download them from this Bitbucket Repository.

Method 2 - CTRL+BREAK (only if Stash is running in a console window)

1. 1. In the Command Console window where Stash is running, open the properties dialog box by right clicking on the title bar and select "Properties".
   2. [Layout] タブを選択します。
   3. Under Screen Buffer Size, set the Height to 3000.
   4. [OK] をクリックします。
   5. 開いているコマンド コンソール上で、CTRL-BREAK を押すと、コマンド コンソールにスレッド ダンプが出力されます。
   6. "Full thread dump" と書かれた行まで、コマンド コンソールをスクロールします。
   7. タイトルバーを右クリックし、編集 ->マークを選択します。スレッドダンプのテキスト全体をハイライト表示します。
   8. Right click the title bar and select Edit -> Copy. The thread dump can then be pasted into a text file.

Method 3 - jstack:

JDK と‬ jstack ツールが合体して、スレッドダンプを発生します。

1. 1. Identify the process. Launch the task manager by, pressing Ctrl + Shift + Esc and find the Process ID of the Java (JIRA) process. You may need to add the PID column using View -> Select Columns ...
   2. ‪‪jstack <pid>‬ を実行して、 単一のスレッドダンプを取得します。このコマンドで、プロセス ID ‪ <pid>‬ の1個のスレッドダンプが取得出来ます。この場合のプロセス ID は22668です。

C:\Users\Administrator>jstack.exe -l 22668 > threaddump.txt

このコマンドにより、現在のディレクトリに threaddump.txt というファイルが出力されます。

Method 4 - VisualVM:

VisualVM is only provided with the JDK distribution so you must have this installed regardless of the scenario describing your environment below. You don't have to change the Java version used by the application to use VisualVM but you will have to add JMX parameters if this is the case or if you are running Stash as a service.

Scenario 1: If Stash is using the JDK and Stash is running as a console application:

1. 1. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
   2. Select the tomcat process in the left-hand pane under "Local".
   3. Select the "Threads" tab in the right pane.

   4. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

Scenario 2: If Stash is not using the JDK and Stash is running as a console application:

1. 1. Stop Stash.

   2. Edit the <Stash installation directory>\bin\setenv.bat and add the following JMX parameters to those already defined for the JVM_SUPPORT_RECOMMENDED_ARGS property. You must choose a free port for JMX to bind to – the following example uses port 7995:

      rem
      rem Occasionally Atlassian Support may recommend that you set some specific JVM arguments.  You can use this variable
      rem below to do that.
      rem
      set JVM_SUPPORT_RECOMMENDED_ARGS=-Dcom.sun.management.jmxremote.port=7995 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

   3. Restart Stash.
   4. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
   5. Select "File" > "Add JMX Connection".
   6. Type: localhost:7995
   7. Select the tomcat process in the left-hand pane under "Local".
   8. Select the "Threads" tab in the right pane.

   9. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).

Scenario 3: If Stash is running as a service:

1. 1. Stop Stash.

   2. Edit the existing service – the default service name is AtlassianStash: <Stash installation directory>\bin\tomcat8w.exe //ES//AtlassianStash  and add the JMX parameters to the service configuration on the "Java" tab under "Java Options" in addition to what is already there and apply the changes. You must choose a free port for JMX to bind to – the following example uses port 7995:

      -Dcom.sun.management.jmxremote.port=7995
      -Dcom.sun.management.jmxremote.authenticate=false
      -Dcom.sun.management.jmxremote.ssl=false

   3. Restart Stash.

   4. Start VisualVM: <JDK installation directory>\bin\jvisualvm.exe
   5. Select "File" > "Add JMX Connection".
   6. Type: localhost:7995
   7. Select the tomcat process in the left-hand pane under "Local".
   8. Select the "Threads" tab in the right pane.

   9. Click the button "Thread Dump" (and then select the "Threads" tab to get back to the button).
      
      

Generating a Thread Dump on Linux, including Solaris and other Unixes

Method 1 - jstack:

Sun JDK 1.5 and above ship with native tool called jstack to perform thread dump.

• • Find the process ID of Stash the JVM using the ps command:

STASH_PID=`ps aux | grep -i stash | grep -i java | awk  -F '[ ]*' '{print $2}'`;

• • Then run the following command 6 times with a 10 second interval.

top -b -H -p $STASH_PID -n 1 > stash_cpu_usage.`date +%s`.txt; jstack $STASH_PID > stash_threads.`date +%s`.txt

• • Look in the resulting cpu usage files to identify which threads are consistently using a lot of CPU time.

  • Take the PID of the top 10 threads which are using CPU time and convert them to Hexadecimal. Eg: 11159 becomes 0x2b97
    
  • Search up the Hex values in the thread dumps to figure out which threads are using up all the CPU

Method 2 - kill signal:

An alternative to the method above is:

• • Find the process ID of Stash the JVM using the ps command:
    
    

ps aux | grep -i stash | grep -i java | awk  -F '[ ]*' '{print $2}'

• • Then run the following command 6 times with a 10 second interval:
    
    

kill -3 <pid>

Note: This will not kill your server (so long as you included the "-3" option, no space in between).
The thread dump will be printed to Stash's standard output (catalina.out).
If you have trouble generating the thread dumps with this method, then use the method "Generating a Thread dump for Windows" as they can also apply for linux, etc.

Thread Dump Tools for analyzing the data

• Samurai
• Thread Dump Analyzer TDA TDA 1.0 Final can be obtained from the java.net