Backup client restore fails due to Out of Memory or GC overhead limit exceeded
症状
The backup client fails during the restore due to the default Java heap space not being enough for the operation to be performed.
One of the logs below can be shown in the atlassian-bitbucket-YYYY-MM-DD.log:
2015-03-09 15:50:58,740 INFO [main] c.a.s.i.b.c.DefaultRestoreClient Processed 726000 changes of 1293334
2015-03-09 15:53:26,155 INFO [main] c.a.s.i.b.c.DefaultRestoreClient Processed 727000 changes of 1293334
2015-03-09 15:53:54,477 ERROR [main] c.a.s.i.backup.client.RestoreMain test-bitbucket-20150304-021150-568.tar could not be restored. Reason: Failed to execute change: Insert Row; nested exception is java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
com.atlassian.bitbucket.internal.backup.liquibase.LiquibaseChangeExecutionException: Failed to execute change: Insert Row; nested exception is java.sql.SQLException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseDao.insert(DefaultLiquibaseDao.java:274) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DatabaseUpdater.endElement(DatabaseUpdater.java:81) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:1.7.0_71]
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) ~[na:1.7.0_71]
at com.atlassian.security.xml.RestrictedXMLReader.parse(RestrictedXMLReader.java:103) ~[atlassian-secure-xml-3.2.4.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseMigrationDao.parse(DefaultLiquibaseMigrationDao.java:206) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseDao.withLock(DefaultLiquibaseDao.java:301) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.liquibase.DefaultLiquibaseMigrationDao.restore(DefaultLiquibaseMigrationDao.java:168) ~[bitbucket-dao-impl-3.5.0.jar:na]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restoreFile(DefaultRestoreClient.groovy:309) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.withDataSource(DefaultRestoreClient.groovy:266) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restoreDatabase(DefaultRestoreClient.groovy:218) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restoreSystem(DefaultRestoreClient.groovy:252) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.DefaultRestoreClient.restore(DefaultRestoreClient.groovy:83) ~[bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.spring.SpringMain.run(SpringMain.groovy:28) [bitbucket-backup-core-1.6.0.jar:na]
at com.atlassian.bitbucket.internal.backup.client.RestoreMain.main(RestoreMain.groovy:55) [bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.JavaVersionCheckingShim.invokeClientClassMain(JavaVersionCheckingShim.java:71) [bitbucket-restore-client.jar:1.6.0]
at com.atlassian.bitbucket.internal.backup.client.JavaVersionCheckingShim.run(JavaVersionCheckingShim.java:85) [bitbucket-restore-client.jar:1.6.0]
... 100 frames suppressedまたは
2015-03-10 14:01:29,052 INFO Processed 806000 changes of 1293334
2015-03-10 14:04:55,066 INFO Processed 807000 changes of 1293334
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.lang.reflect.Method.copy(Method.java:151)
at java.lang.reflect.ReflectAccess.copyMethod(ReflectAccess.java:136)
at sun.reflect.ReflectionFactory.copyMethod(ReflectionFactory.java:300)
at java.lang.Class.copyMethods(Class.java:2934)
at java.lang.Class.getDeclaredMethods(Class.java:1860)
at org.codehaus.groovy.reflection.CachedClass$3$1.run(CachedClass.java:84)
at java.security.AccessController.doPrivileged(Native Method)
at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:81)
at org.codehaus.groovy.reflection.CachedClass$3.initValue(CachedClass.java:79)
at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:46)
at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:37)
at org.codehaus.groovy.reflection.CachedClass.getMethods(CachedClass.java:250)
at org.codehaus.groovy.reflection.CachedMethod.find(CachedMethod.java:58)
at groovy.lang.MetaClassImpl.applyPropertyDescriptors(MetaClassImpl.java:2474)
at groovy.lang.MetaClassImpl.setupProperties(MetaClassImpl.java:2250)
at groovy.lang.MetaClassImpl.addProperties(MetaClassImpl.java:3255)
at groovy.lang.MetaClassImpl.initialize(MetaClassImpl.java:3226)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:210)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241)
at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:251)
at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:259)
at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:855)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.createPojoSite(CallSiteArray.java:122)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallSite(CallSiteArray.java:163)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
at com.atlassian.bitbucket.internal.backup.client.liquibase.LoggingLiquibaseRestoreMonitor.onAppliedChange(LoggingLiquibaseRestoreMonitor.groovy:26)
at com.atlassian.bitbucket.internal.backup.liquibase.DatabaseUpdater.endElement(DatabaseUpdater.java:83)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1782)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2973)原因
The default MaxHeapSize defined for the Java in your system is not enough for performing the restore. On the output below, for example, this value is 1GB:
$ java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
uintx AdaptivePermSizeWeight = 20 {product}
intx CompilerThreadStackSize = 0 {pd product}
uintx ErgoHeapSizeLimit = 0 {product}
uintx HeapSizePerGCThread = 87241520 {product}
uintx InitialHeapSize := 62733632 {product}
uintx LargePageHeapSizeThreshold = 134217728 {product}
uintx MaxHeapSize := 1004535808 {product}
uintx MaxPermSize = 174063616 {pd product}
uintx PermSize = 21757952 {pd product}
intx ThreadStackSize = 1024 {pd product}
intx VMThreadStackSize = 1024 {pd product}
java version "1.7.0_71"
OpenJDK Runtime Environment (rhel-2.5.3.1.el6-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)ソリューション
While executing the restore, add a parameter to the restore command, increasing the heap size for that execution by passing a higher maxHeapSize parameter. For this specific example, we raised it to -Xmx2g.
According to the link above, that's how you'd normally do:
$ java -Djdbc.override=true -Djdbc.driver=org.postgresql.Driver -Djdbc.url=jdbc:postgresql://HOSTNAME:PORT/DATABASE -Djdbc.user=bitbucketuser -Djdbc.password=password -Dbitbucket.home="path/to/bitbucket/home" -jar /path/to/bitbucket-restore-client.jar /path/to/original/backup/fileChange the command below into:
$ java -Xmx2g -Djdbc.override=true -Djdbc.driver=org.postgresql.Driver -Djdbc.url=jdbc:postgresql://HOSTNAME:PORT/DATABASE -Djdbc.user=bitbucketuser -Djdbc.password=password -Dbitbucket.home="path/to/bitbucket/home" -jar /path/to/bitbucket-restore-client.jar /path/to/original/backup/file
最終更新日 2016 年 4 月 6 日
Powered by Confluence and Scroll Viewport.