Application crashes with SIGSEGV jvm.dll ciObjectFactory::create_new_metadata
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
問題
Java application (including Atlassian products) may sporadically crash with the following message appearing in the logs.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f9026d3cbd7, pid=6999, tid=140252587140864
#
# JRE version: Java(TM) SE Runtime Environment (8.0_60-b27) (build 1.8.0_60-b27)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V [libjvm.so+0x436bd7] ciObjectFactory::create_new_metadata(Metadata*)+0x327
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00007f9022889800): JavaThread "C2 CompilerThread4" daemon [_thread_in_vm, id=7173, stack(0x00007f8f1992a000,0x00007f8f19a2b000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000013
診断
環境
- Running on Java 8 update 40 or later (1.8.0_40+)
- Have a Java agent enabled for monitoring the JVM
Diagnostic Steps
- You can verify your Java version in every Atlassian product on the Administration section, navigating to the Troubleshooting and Support Tools >> System Information tab, under the heading Java Runtime Environment
- Your JVM startup arguments should contain a Java agent, see the following example highlighting it:
- -Xmx8g -Xms8g -XX:MaxMetaspaceSize=512m -Djava.awt.headless=true -XX:+UseCodeCacheFlushing -XX:+UseG1GC -XX:+UseNUMA -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:/usr/local/atlassian/confluence/current/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=50M -Dcom.atlassian.confluence.extra.calendar3.jira.timeout.socket=30000 -javaagent:/usr/local/newrelic/current/newrelic.jar
The above arguments can be checked in the
<Bitbucket-installation-directory>/bin/_start-webapp.sh
file or in theapplication.xml
file in the Support Zip shared by the customer.
原因
The cause is not yet identified, there are a number of Java bugs that may be related to the issue (listed below). What we have identified so far that it does not only affect Atlassian applications, but Java application running Java 8 update 40 or later in general, in combination with Java agent(s) using Spring AOP instrumentation. Although it may not be the only possible way to reproduce the issue, what we have seen in all issues reported to us so far, producing these symptoms is that all instances affected were having a Java agent enabled.
So far we have encountered the crash with the following Java agents (however the affected ones may not be limited to these):
- AppDynamics
- Customer reports state that modifying the agent configurations can work around the problem that is related to instrumentation (for JIRA):
Agent configuration change: edit the file
AgentDir/ver**/conf/app-agent-config.xml
and add the following snippet into<fork-config>
element.<excludes filter-type="STARTSWITH" filter-value="com.atlassian,net.sf.ehcache.Cache"/>
Controller changes: From the Controller UI go to
Configure -> Instrumentation -> Callgraph Settings
and Uncheck theEnable Aggressive Snapshot Collection
.Node Property: exclude-interceptors Type: String Value: com.singularity.tm.TurboCustomExitPointInterceptor
Node Property: disable-custom-exit-points-for Type: String Value: EHCache
Node Property: disable-ootb-turbo-interceptors Type: Boolean Value: true
- Customer reports state that modifying the agent configurations can work around the problem that is related to instrumentation (for JIRA):
- NewRelic
- Customer reports state, that version 3.17.0 contains a fix that eliminated the problem: The Spring AOP instrumentation is now disabled by default. It was exercising a bug in certain versions of Java 8 that caused the JVM to crash)
回避策
Please note that the workaround you apply is done at your own risk, you need to evaluate the possible impacts of either.
- Disable/remove Java agent from JVM startup arguments
- See the agent-specific possible workarounds above
- Contact your Java agent vendor
ソリューション
Contact the vendor of the Java agent you use and inquire about this bug and see if they already have a version with a fix available.
Watch the following Java bugs and monitor the progress to see when a version with the fix for it is released.
- Crash in HotSpot with jvm.dll+0x42b48 ciObjectFactory::create_new_metadata
- Fatal error in ciObjectFactory::create_new_metadata with Java8u40
参考