Server Unreachable - Timeout when merging or viewing Diff of pull request

症状

Upon merging a pull request the merge dialog will be displayed for around 30 seconds before timing out with "Server Unreachable – The server didn't respond. You may retry your request when the server comes back up." Note that refreshing the page in the browser actually shows that the pull request was merged. The "Delete Branch" option doesn't work and the timeout message is also seen when viewing the diff of a pull request. Also, pushing code via command-line gets to the point where all objects are written 100% and then pauses for ~45 seconds before finally completing.

Message seen on the screen:

原因

Browser-side code will timeout after one minute. The merge completes because the HTTP POST leaves the customer browser after clicking "Merge". It arrives at the Stash server which processes it but the response to the POST is lost on the way back to the client. Possible causes:

• Proxy overload
• Network issues
• Stash running slow on a Windows server because it is being executed as a user that is part of the Administrator group.
• Plugins in your browser (i.e. reported similar issues proved to have been caused by AdBlock plugin)
• Add-ons on your Stash instance (i.e. reported similar issues proved to have been caused by Awesome Graphs for Stash plugin)
• Git 1.8.3.X and Git 2.2.x - 2.4.0 working with a STASH_HOME mounted on a NFS are not supported by Stash due to performance issues. Please consult our Supported platforms document.
• STASH-7360 - Getting issue details... STATUS : if you have a large repository containing binary files merges often genuinely take longer than a minute. The error "Server Unreachable" is given even though the merge is still occurring behind the scenes.
• STASH-7457 - Getting issue details... STATUS : after upgrading to Stash 3.9.X some pull requests hang for long periods trying to display the diff. When the diff is eventually displayed, the pull request indicates the diff has conflicts but none are shown. This can also manifest as a lengthy (~5 minute) delay merging pull requests, after which the pull request ultimately fails to merge. A "Server unreachable" error will be shown after ~1 minute due to STASH-7360, but refreshing shows that the merge didn't ever complete.

診断

• Bypass the reverse proxy to try and fix the problem.
  
• Check you network latency.
• Check your browser:
  • Test access with different browsers/versions
  • Run your browser in "safe mode" to isolate it from installed plugins that could be causing the issue (i.e.how to start firefox in "safe mode")
  • Check your browser configuration for a proxy that could be causing the issue. Bypass this proxy.
• Check your logs for Stash Plugin Issues

  • Awesome Graphs: check for messages like the ones below:

    2014-05-09 00:08:10,479 DEBUG [pool-8-thread-1]  com.atlassian.stash.scm.BaseCommand Executed /usr/local/bin/git log 3472f5f866c30e31758853016401189e3b02b6d2 --pretty=format:%H --shortstat --no-merges
    2014-05-09 00:08:10,479 WARN  [pool-8-thread-1]  c.s.s.g.s.LogThrowableExitHandler Unable to execute command completely: /usr/local/bin/git log 3472f5f866c30e31758853016401189e3b02b6d2 --pretty=format:%H --shortstat --no-merges. Throwable: com.atlassian.utils.process.ProcessTimeoutException: process timed out
    2014-05-09 00:08:10,479 WARN  [pool-8-thread-1]  c.s.s.g.idx.GraphsIndexingService ((Diffs)) -> 359
    2014-05-09 00:08:10,551 WARN  [pool-8-thread-1]  c.s.s.graphs.idx.GraphsIndexingJob GraphsIndexingJob --> stop()
    2014-05-09 00:08:10,551 WARN  [pool-8-thread-1]  c.s.s.graphs.idx.GraphsIndexingJob GraphsIndexingJob --> start()
    2014-05-09 00:08:26,898 DEBUG [pool-8-thread-1]  com.atlassian.stash.scm.BaseCommand Executed /usr/local/bin/git rev-list --format=%H%x02%h%x02%P%x02%p%x02%aN%x02%aE%x02%at%n%B%n%x03 --stdin --
    2014-05-09 00:08:26,900 WARN  [pool-8-thread-1]  c.s.s.g.idx.GraphsIndexingService (()) -> 8592

• To see if the problem is being caused by "Git 1.8.3.X and Git 2.2.x - 2.4.0" or  STASH-7360 - Getting issue details... STATUS  or  STASH-7457 - Getting issue details... STATUS
  • Enable your Stash Profiling logging

    • Look for the following profiling event:

      2015-06-01 11:37:47,273 | drift:thread-1 | - | - | -
      [121430ms] - Drift: Calculate for com.atlassian.stash.internal.pull.comment.drift.DriftCommentUpdateProcessor$CommentDriftCalculator@76cec2d0
        [121368ms] - Drift: Map anchors 3193
          [16ms] - Set com.atlassian.stash.internal.pull.comment.CommentPostProcessor.processAll(InternalCommentable,Set)
            [16ms] - com.atlassian.stash.internal.comment.PluginCommentPostProcessor.processAll 1 comments
              [16ms] - com.atlassian.stash.internal.comment.like.property.CommentLikePropertyProvider.provideProperties for 1 comments
          [121336ms] - git: with pull request lock 2:2533
            [15ms] - InternalPullRequest com.atlassian.stash.internal.pull.PullRequestDao.findByRepositoryScopedId(int,long)
            [121321ms] - git: resolve effective diff 2533@1
              [120744ms] - git: create new merge 2533@1
                [120681ms] - git: merge <USERNAME>/<REPOSITORY>:refs/heads/client_seeding_cleanup -> <PROJECT>/<REPOSITORY>:refs/heads/development
                  [172ms] - git: clone 2
                    [172ms] - C:\Program Files (x86)\Git\cmd\git.exe clone --no-checkout --shared -b development D:\Atlassian\stash-home\shared\data\repositories\2 D:\Atlassian\stash-home\tmp\git\<REPOSITORY>-work7276869962633828049.git
                  [140ms] - git: reset 818beae5e0bb5faa3998e7fe5bce3b64f2318185
                    [140ms] - C:\Program Files (x86)\Git\cmd\git.exe reset --quiet 818beae5e0bb5faa3998e7fe5bce3b64f2318185 --
                  [328ms] - git: merge 57fab50d40b6286ae12f20d8cfb4d96c3f47945b
                    [328ms] - C:\Program Files (x86)\Git\cmd\git.exe merge -m Automatic merge --no-ff --log 57fab50d40b6286ae12f20d8cfb4d96c3f47945b
                  [120041ms] - git: fetch 2
                    [120041ms] - C:\Program Files (x86)\Git\cmd\git.exe pack-objects --delta-base-offset -q --stdout
              [514ms] - git: resolve merge-base 2533@1
                [483ms] - git: fetch client_seeding_cleanup for 2533@1
                  [483ms] - C:\Program Files (x86)\Git\cmd\git.exe fetch D:\Atlassian\stash-home\shared\data\repositories\183 refs/heads/client_seeding_cleanup:
                [31ms] - C:\Program Files (x86)\Git\cmd\git.exe merge-base 57fab50d40b6286ae12f20d8cfb4d96c3f47945b 818beae5e0bb5faa3998e7fe5bce3b64f2318185
              [47ms] - git: update refs 2533@1
                [32ms] - C:\Program Files (x86)\Git\cmd\git.exe update-ref stash-refs/pull-requests/2533/from 57fab50d40b6286ae12f20d8cfb4d96c3f47945b 3ea38ec83a47bb90eaca5774e062c561e296ffd3
                [15ms] - C:\Program Files (x86)\Git\cmd\git.exe update-ref stash-refs/pull-requests/2533/merge 818beae5e0bb5faa3998e7fe5bce3b64f2318185
        [62ms] - Drift: Strategy - Composite (FileComment, FastForward, Diff, OrphanAll) 3193
          [31ms] - Drift: Strategy - FastForward 3193
            [31ms] - Drift: FastForward - Merge changes [3ea38ec83a47bb90eaca5774e062c561e296ffd3]->[57fab50d40b6286ae12f20d8cfb4d96c3f47945b] 3193
              [31ms] - C:\Program Files (x86)\Git\cmd\git.exe diff-tree -C -r --format=%H 3ea38ec83a47bb90eaca5774e062c561e296ffd3 57fab50d40b6286ae12f20d8cfb4d96c3f47945b --
          [31ms] - Drift: Strategy - Diff 3193
            [31ms] - Drift: Diff - Merge diff [3ea38ec83a47bb90eaca5774e062c561e296ffd3]->[57fab50d40b6286ae12f20d8cfb4d96c3f47945b] 3193
              [31ms] - C:\Program Files (x86)\Git\cmd\git.exe diff -C --color=never -U0 --dst-prefix=dst:// --src-prefix=src:// 3ea38ec83a47bb90eaca5774e062c561e296ffd3 57fab50d40b6286ae12f20d8cfb4d96c3f47945b --

  • Git 1.8.3.X
    

    • If the operation is taking a long time on the [172ms] - git: clone line, that's likely Git 1.8.3

  • STASH-7360 - Getting issue details... STATUS
    • If the operation is taking a long time on the [328ms] - git: merge line, that's STASH-7360
  
  

  • STASH-7457 - Getting issue details... STATUS

    • If the operation is taking a long time on the [120041ms] - git: fetch line, that's STASH-7457
      
      
  • Git 2.2-2.4.0 on NFS
    
    • A long wait on the "git: merge" line might also mean Git 2.2-2.4.0 on NFS

ソリューション

For each of the causes above, the solution would be:

• Correctly configure the timeout on the proxy server or fix its overload.
• Fix any latency issue that could be happening on your network.
• Apply the recommendations described in Git push operations extremely slow on Windows.
• Uninstall offending browser plugin.
• Uninstall offending Stash add-on. Instructions: Uninstalling apps
• Upgrade your Git. Instructions: Installing and upgrading Git
• Upgrade to a release that fixes STASH-7360 - Getting issue details... STATUS .
• Upgrade to a release that fixes STASH-7457 - Getting issue details... STATUS .