Bitbucket Data Center node fails to join cluster - Current node is unable to service requests

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server と Data Center のみ - この記事は、サーバーおよびデータセンター プラットフォームのアトラシアン製品にのみ適用されます。

問題

Your newly provisioned Bitbucket Server node fails to join the cluster:

atlassian-bitbucket.log
2016-11-07 11:33:41,179 WARN  [hz.hazelcast.IO.thread-Acceptor]  com.hazelcast.nio.tcp.SocketAcceptor [10.137.158.47]:5701 [testbitbucket] [3.5.2-atlassian-37] com.atlassian.stash.internal.cluster.NodeConnectionException: Required property 'sharedHome' should be '/storage/data/bitbucket/shared' but is '/storage/data/bitbuckethome/shared', Required property 'timeZone' should be 'UTC' but is 'Australia/Victoria'
com.atlassian.stash.internal.cluster.NodeConnectionException: Required property 'sharedHome' should be '/storage/data/bitbucket/shared' but is '/storage/data/bitbuckethome/shared', Required property 'timeZone' should be 'UTC' but is 'Australia/Victoria'
        at com.atlassian.stash.internal.cluster.DefaultClusterJoinManager.negotiateOutcome(DefaultClusterJoinManager.java:254) ~[bitbucket-service-impl-4.10.1.jar:na]
        at com.atlassian.stash.internal.cluster.DefaultClusterJoinManager.accept(DefaultClusterJoinManager.java:99) ~[bitbucket-service-impl-4.10.1.jar:na]
        at com.atlassian.stash.internal.hazelcast.ClusterJoinSocketInterceptor.onAccept(ClusterJoinSocketInterceptor.java:44) ~[bitbucket-service-impl-4.10.1.jar:na]
        at com.hazelcast.nio.tcp.TcpIpConnectionManager.interceptSocket(TcpIpConnectionManager.java:165) ~[hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
        at com.hazelcast.nio.tcp.SocketAcceptor.configureAndAssignSocket(SocketAcceptor.java:142) [hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
        at com.hazelcast.nio.tcp.SocketAcceptor.acceptSocket(SocketAcceptor.java:128) [hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
        at com.hazelcast.nio.tcp.SocketAcceptor.run(SocketAcceptor.java:70) [hazelcast-3.5.2-atlassian-37.jar:3.5.2-atlassian-37]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
        ... 20 frames trimmed

原因

When joining the cluster, a set of parameters need to be the same between the node. That's a requirement.

In the stack trace above, the shared directory and the TimeZone diverge between nodes.

The timezone of the affected node does not match that of the cluster:

atlassian-bitbucket.log
2015-05-07 17:59:42,508 DEBUG [hz.hazelcast.cached.thread-1]  c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Starting to connect to Address[192.168.1.3]:5701
2015-05-07 17:59:42,509 DEBUG [hz.hazelcast.cached.thread-1]  c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Connecting to /192.168.1.3:5701, timeout: 0, bind-any: true
2015-05-07 17:59:42,509 DEBUG [hz.hazelcast.cached.thread-1]  c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Successfully connected to: Address[192.168.1.3]:5701 using socket Socket[addr=/192.168.1.3,port=5701,localport=52601]
2015-05-07 17:59:42,533 DEBUG [hz.hazelcast.cached.thread-1]  c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Could not connect to: /192.168.1.3:5701. Reason: NodeConnectionException[ 'sun.util.calendar.ZoneInfo[id="US/Pacific-New",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific-New,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]' but is 'sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]']
2015-05-07 17:59:42,534 DEBUG [hz.hazelcast.cached.thread-1]  c.hazelcast.nio.tcp.SocketConnector [192.168.1.2]:5701 [seo-bitbucket-beta] [3.3-RC3] Required property 'timeZone' should be 'sun.util.calendar.ZoneInfo[id="US/Pacific-New",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific-New,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]' but is 'sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]'
com.atlassian.bitbucket.internal.cluster.NodeConnectionException: Required property 'timeZone' should be 'sun.util.calendar.ZoneInfo[id="US/Pacific-New",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific-New,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]' but is 'sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]'
    at com.atlassian.bitbucket.internal.cluster.DefaultClusterJoinManager.checkNoRequirementErrors(DefaultClusterJoinManager.java:284) ~[DefaultClusterJoinManager.class:na]
    at com.atlassian.bitbucket.internal.cluster.DefaultClusterJoinManager.checkRequirementsMatchServer(DefaultClusterJoinManager.java:213) ~[DefaultClusterJoinManager.class:na]
    at com.atlassian.bitbucket.internal.cluster.DefaultClusterJoinManager.joinServer(DefaultClusterJoinManager.java:81) ~[DefaultClusterJoinManager.class:na]
    at com.atlassian.bitbucket.internal.hazelcast.ClusterJoinSocketInterceptor.onConnect(ClusterJoinSocketInterceptor.java:48) ~[ClusterJoinSocketInterceptor.class:na]
    at com.hazelcast.nio.tcp.TcpIpConnectionManager.interceptSocket(TcpIpConnectionManager.java:155) ~[TcpIpConnectionManager.class:3.3-RC3]
    at com.hazelcast.nio.tcp.SocketConnector.tryToConnect(SocketConnector.java:131) ~[SocketConnector.class:3.3-RC3]
    at com.hazelcast.nio.tcp.SocketConnector.run(SocketConnector.java:67) ~[SocketConnector.class:3.3-RC3]
    at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:209) [CachedExecutorServiceDelegate$Worker.class:3.3-RC3]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09-icedtea]
    at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09-icedtea]
    at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59) [PoolExecutorThreadFactory$ManagedThread.class:3.3-RC3]
    ... 1 frame trimmed

ソリューション

Update the node's time zone to match that of the cluster by symlinking /etc/localtime to the correct time zone file usually found here: /usr/share/zoneinfo

How to identify there is still a JVM mismatch even after the symlink above?

If the node's time zone matches the other nodes in the cluster but Bitbucket Server still reports a mismatch, you have force the JVM to acknowledge the timezone you want by passing an additional parameter to the JVM (Java Virtual Machine).

 How to identify the cause for the persistent mismatch?

  • You can checkout the repository below and run the jar program to output the Time Zone on each one of your nodes. The code can be found here. Use the procedure below:

    Checkout the code into your node:
    $ git clone git@bitbucket.org:thiago_bomfim/gettimezone.git
    $ cd gettimezone/target
    List the process running Bitbucket server: 
    ps -ef | grep bitbucket | grep -v elastic
    atlbitb+   843     1  0 Aug23 ?        00:14:40 /usr/java/jre1.8.0_73/bin/java -Djava.util.logging.config.file=/opt/atlassian/bitbucket/4.7.1/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx768m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Datlassian.standalone=BITBUCKET -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -Dorg.apache.catalina.connector.Response.ENFORCE_ENCODING_IN_GET_WRITER=false -Djava.library.path=/opt/atlassian/bitbucket/4.7.1/lib/native:/var/atlassian/application-data/bitbucket/lib/native -Dbitbucket.home=/var/atlassian/application-data/bitbucket -Djava.endorsed.dirs=/opt/atlassian/bitbucket/4.7.1/endorsed -classpath /opt/atlassian/bitbucket/4.7.1/bin/bitbucket-bootstrap.jar:/opt/atlassian/bitbucket/4.7.1/bin/bootstrap.jar:/opt/atlassian/bitbucket/4.7.1/bin/tomcat-juli.jar -Dcatalina.base=/opt/atlassian/bitbucket/4.7.1 -Dcatalina.home=/opt/atlassian/bitbucket/4.7.1 -Djava.io.tmpdir=/opt/atlassian/bitbucket/4.7.1/temp com.atlassian.stash.internal.catalina.startup.Bootstrap start
    Use the Java running this process to output the timezone of your node. From the example above:
    /usr/java/jre1.8.0_73/bin/java -jar gettimezone-1.0-SNAPSHOT.jar
    The Timezone of this JVM is: "Australia/Sydney"
  • Alternatively to the method above, if you generate a Support Zip for each one of the nodes in the cluster (Administration -> Atlassian Support Tools -> Support Zip -> Create) and look into the section below, chances are you will see the mismatch:
Node 1
  <java-runtime-environment>
  ...
     <user.timezone>America/Montreal</user.timezone>
  ...
  </java-runtime-environment>
Node 2
  <java-runtime-environment>
  ...
      <user.timezone>America/Chicago</user.timezone>
  ...
  </java-runtime-environment>


How to force the JVM with the expected TZ you used in the symlink?

The following example sets the time zone for Chicago. You have to do that for each one of the nodes:

  •  Edit <BITBUCKET_INSTALL>/bin/setenv.sh:
<BITBUCKET_INSTALL>/bin/setenv.sh
-Duser.timezone=America/Chicago

setenv and environment variable changes in Bitbucket Server 5.0+

Starting with Bitbucket Server 5.0, setenv.sh and setenv.bat have been removed. The options that were set in this file can now be set via environment variables. Where to set the environment variable depends on which Operating System you're running on.

Linux

When using the atlbitbucket service on Linux, the environment variables are ignored. You must set the parameters in _start-webapp.sh. These values will be read when the service starts.

As an example, to set JVM_SUPPORT_RECOMMENDED_ARGS, you would add this line to the file:

JVM_SUPPORT_RECOMMENDED_ARGS=-XX:+HeapDumpOnOutOfMemoryError

Windows

Set the parameter as an environment variable for the user running Bitbucket Server. For example, if you want to set JVM_SUPPORT_RECOMMENDED_ARGS, create it as an environment variable and assign the appropriate value to it. When Bitbucket Server starts using the startup scripts or service, it will pick up and apply this value.

Note that after applying the configuration above, a restart on your nodes is required so the node will pick up the JVM argument (i.e. after the restart you should see this parameter on the output of ps -ef|grep bitbucket)


The List of TZ Database Time Zones documentation on Wikipedia contains a comprehensive list of possible time zones that can be set.

Examples of common time zones

タイム ゾーンコー​​ド
シドニーAustralia/Sydney
Los AngelesAmerica/Los_Angeles
New YorkAmerica/New_York
パリEurope/Paris
RomeEurope/Rome
UTCUTC
MadridEurope/Madrid


How to change BTIBUCKET_HOME?

 Your installer would have set that in these 2 different files (depending on your operating system):

  • <Bitbucket_Install>/bin/set-bitbucket-home.sh or
  • <Bitbucket_Install>/bin/set-bitbucket-home.bat

For Linux, <Bitbucket_Install> is set to /opt/atlassian/bitbucket/4.10.1/ by default.

最終更新日 2017 年 5 月 4 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する
Powered by Confluence and Scroll Viewport.