JVM is not reachable with jstat and jstack

Platform Notice: Data Center Only - This article only applies to Atlassian products on the Data Center platform.

Note that this KB was created for the Data Center version of the product. Data Center KBs for non-Data-Center-specific features may also work for Server versions of the product, however they have not been tested. Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Except Fisheye and Crucible

Summary

Generating Jira thread dumps using "jstack" returns error "Unable to open socket file"

Environment

Jira Software/Service Management running on Linux hosts with Java versions 8 or 11

Diagnosis

Trying to generate Jira thread dumps using jstack, the command returns the following error:

1
2

$ sudo -u jira /usr/lib/jvm/jdk-11.0.2+7/bin/jstack 1455
1455: Unable to open socket file /proc/1455/root/tmp/.java_pid1455: target process 1455 doesn't respond within 10500ms or HotSpot VM not loaded

The path "/proc/<PID>/root" is a symbolic link to the root path ( / )

1
2

ls -l /proc/<PID>/root
lrwxrwxrwx 1 jira jira 0 dez  3 18:29 /proc/<PID>/root -> /

So the path "/proc/<PID>/root/tmp/" is pointing to "/tmp" to store temporary files for the processes.

Checking the "/tmp", the Java socket file was not present:

1
2

ls -l /tmp/.java_pid<PID>
ls: cannot access '/tmp/.java_pid<PID>': No such file or directory

Cause

There are some external factors that can cause the socket file to be deleted from "/tmp":

External routine tools to clean files older than X days;
Linux "systemd-tmpfiles-clean" removing Java Hooks
(More information can be found in Red Hat Bugzilla – Bug 1704608 but this is not limited to RedHat)
Manual removal of the files from "/tmp"

Solution

1 - If any external tool is being used to delete old files from "/tmp", configure the tool to ignore the patterns below:

1
2

/tmp/hsperfdata_*
/tmp/.java_pid*

2 - If it is found to be the Linux Systemd Cleanup task deleting the files, do the following:

Edit the file "/usr/lib/tmpfiles.d/tmp.conf" and add the 2 lines below:

(The "x" in both lines are to indicate that the patterns must be excluded from the cleanup task)

1
2
3

# Exclude java-hooks
x /tmp/hsperfdata_*
x /tmp/.java_pid*

Workaround

If the socket file was deleted and for a troubleshooting situation it is necessary to generate thread dumps before restarting the application, the command below can be used to send thread dumps to "Catalina.out" log.

See more details in the document Generating a thread dump under the section "Linux/Unix/OS X environment" > "Linux/Unix command line"

kill -3 <PID>