Clone fails with "exited with code -1,002" error when Datadog Java agent is embedded in JVM
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
要約
Clone operations will intermittently fail to complete when using Datadog Java agent to monitor application stats.
環境
- Bitbucket 7.5.X
- Datadog Java agent 7.2.0 with dd-trace-java.jar 0.64.0
診断
When the clone operation fails the following messages are found in the git client machine and in Bitbucket logs.
Git client log
git clone ssh://git@<BITBUCKET_BASE_URL>:7999/test_project/Test-repo.git
Cloning into 'test_repo'...
remote: Enumerating objects: 1357, done.
remote: Counting objects: 100% (1357/1357), done.
remote: Compressing objects: 100% (4456/4456), done.
error: git upload-pack: git-pack-objects died with error. KiB/s
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
remote: aborting due to possible repository corruption on the remote side.
fatal: early EOF
fatal: index-pack failed
atlassian-bitbucket.log
2020-10-05 00:08:14,952 WARN [git-hook-callback:thread-2] test_user @1GTT9VBM7x636502x1 1ygpuz7 192.168.X.X,0:0:0:0:0:0:0:1 "POST /scm/test_project/test_repo.git/git-upload-pack HTTP/1.0" c.a.s.i.s.g.h.DefaultGitHookService git-pack-objects: Hook handling failed
com.atlassian.bitbucket.scm.CommandFailedException: '/usr/bin/git pack-objects --revs --thin --stdout --progress --delta-base-offset' exited with code -1,002
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onError(DefaultCommandExitHandler.java:43)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:111)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:285)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.finish(NioNuProcessHandler.java:326)
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.asResult(NioNuProcessHandler.java:241)
at com.atlassian.bitbucket.internal.process.nu.NuNioProcessHelper.run(NuNioProcessHelper.java:77)
at com.atlassian.bitbucket.internal.process.NioCommand.call(NioCommand.java:52)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.streamFromGit(CachingPackObjectsHookHandler.java:273)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.access$000(CachingPackObjectsHookHandler.java:63)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler$2.apply(CachingPackObjectsHookHandler.java:208)
at com.atlassian.util.contentcache.internal.AbstractCacheEntry$BaseAccess.streamToCache(AbstractCacheEntry.java:362)
at com.atlassian.util.contentcache.internal.AbstractCacheEntry$CacheEntryAccess.stream(AbstractCacheEntry.java:452)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.streamFromCache(CachingPackObjectsHookHandler.java:245)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.maybeHandle(CachingPackObjectsHookHandler.java:193)
at com.atlassian.stash.internal.scm.git.hook.handler.CachingPackObjectsHookHandler.handle(CachingPackObjectsHookHandler.java:95)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.doHandleRequest(DefaultGitHookService.java:276)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.handleRequest(DefaultGitHookService.java:233)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService.handleRawRequest(DefaultGitHookService.java:194)
at com.atlassian.stash.internal.scm.git.hook.DefaultGitHookService$1.lambda$run$0(DefaultGitHookService.java:154)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.lang.Thread.run(Thread.java:834)
... 1 frame trimmed
Caused by: com.atlassian.utils.process.ProcessException: Non-zero exit code: -1002
at com.atlassian.bitbucket.internal.process.nu.NioNuProcessHandler.callExitHandler(NioNuProcessHandler.java:277)
... 19 common frames omitted
Followed by
atlassian-bitbucket.log
2020-10-05 00:08:15,321 ERROR [http-scmrequest-handler:thread-8] test_user @1GTT9VBM7x636502x1 1hclqk0 192.168.X.X,0:0:0:0:0:0:0:1 "POST /scm/test_project/test_repo.git/git-upload-pack HTTP/1.0" c.a.s.i.s.g.p.h.GitSmartExitHandler test_project/test_repo[194]: Read request from 192.168.X.X failed: com.atlassian.utils.process.ProcessException: Non-zero exit code: 1
The following was written to stderr:
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption on the remote side.Datadog will also throw the following error for multiple Bitbucket packages.
launcher.log
[dd.trace 2020-10-03 21:15:17:942 -0400] [spring-startup] ERROR datadog.trace.agent.tooling.HelperInjector - Error preparing helpers while processing class com.atlassian.plugins.impl.servlet.AdminServlet for Servlet3Instrumentation. Failed to inject helper classes into instance com.atlassian.plugins.static-assets-url [113]
java.lang.IllegalStateException: Error invoking (accessor)::defineClass
at net.bytebuddy.dynamic.loading.ClassInjector$UsingReflection$Dispatcher$UsingUnsafeInjection.defineClass(ClassInjector.java:922)原因
Datadog Java agent fails to inject helper classes into Bitbucket packages causing git hook to fail and impacting clone operations.
回避策
Remove Datadog JVM parameters, shown below, from the instance and restart Bitbucket;
These parameters are usually set in the <Bitbucket_Install>/bin/_start-webapp.sh
-javaagent:/opt/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.service=<SERVICE_NAME> -Ddd.env=<ENV> -Datlassian.org.osgi.framework.bootdelegation.extra=datadog.*
- If needed, engage with Datadog Support for further details and troubleshooting.
最終更新日: 2020 年 10 月 28 日
Powered by Confluence and Scroll Viewport.