Hipchat Server upgrade stalled and eventually timed out while executing apt-get-update-build-essentials

関連コンテンツ

  • 関連コンテンツがありません

お困りですか?

アトラシアン コミュニティをご利用ください。

コミュニティに質問

プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。

Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.

*Fisheye および Crucible は除く

 

 

問題

When running a Hipchat Server upgrade through the terminal/command-line, admin noticed the upgrade stalled and there is not much progress on the console. The process eventually failed and the following appears in /var/log/chef.log

[2016-11-23T13:28:38+08:00] INFO: Processing execute[apt-get-update-build-essentials] action run (build-essential::debian line 22)
[2016-11-23T14:29:32+08:00] ERROR: Running exception handlers
[2016-11-23T14:29:32+08:00] ERROR: Exception handlers complete
[2016-11-23T14:29:32+08:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
[2016-11-23T14:29:32+08:00] ERROR: execute[apt-get-update-build-essentials] (build-essential::debian line 22) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command execeded allowed execution time, killed by TERM signal.
---- Begin output of apt-get update ----
STDOUT: Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Ign http://ppa.launchpad.net trusty InRelease
Err http://ppa.launchpad.net trusty Release.gpg
  Connection failed
Ign http://ppa.launchpad.net trusty Release
Ign http://ppa.launchpad.net trusty/main amd64 Packages/DiffIndex
Ign http://ppa.launchpad.net trusty/main i386 Packages/DiffIndex
STDERR: 
---- End output of apt-get update ----
Ran apt-get update returned 
[0m
================================================================================[0m
[31mError executing action `run` on resource 'execute[apt-get-update-build-essentials]'[0m
================================================================================[0m

[0m
Mixlib::ShellOut::CommandTimeout[0m
--------------------------------[0m
Command timed out after 3600s:
Command execeded allowed execution time, killed by TERM signal.
---- Begin output of apt-get update ----
STDOUT: Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Ign http://ppa.launchpad.net trusty InRelease
Err http://ppa.launchpad.net trusty Release.gpg
  Connection failed
Ign http://ppa.launchpad.net trusty Release
Ign http://ppa.launchpad.net trusty/main amd64 Packages/DiffIndex
Ign http://ppa.launchpad.net trusty/main i386 Packages/DiffIndex
STDERR: 
---- End output of apt-get update ----
Ran apt-get update returned [0m

[0m
Cookbook Trace:[0m
---------------[0m
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:30:in `from_file'
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/default.rb:21:in `from_file'[0m

[0m
Resource Declaration:[0m
---------------------[0m
# In /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb

 22: execute "apt-get-update-build-essentials" do
 23:   command "apt-get update"
 24:   action :nothing
 25:   # tip: to suppress this running every time, just use the apt cookbook
 26:   not_if do
 27:     ::File.exists?('/var/lib/apt/periodic/update-success-stamp') &&
 28:     ::File.mtime('/var/lib/apt/periodic/update-success-stamp') > Time.now - 86400*2
 29:   end
 30: end.run_action(:run) if node['build_essential']['compiletime']
 31: 
 32: %w{
 33:   autoconf
 34:   binutils-doc
 35:   bison
 36:   build-essential
 37:   flex
 38: }.each do |pkg|
 39: 
 40:   r = package pkg do
 41:     action( node['build_essential']['compiletime'] ? :nothing : :install )
 42:   end
 43:   r.run_action(:install) if node['build_essential']['compiletime']
 44: 
 45: end
[0m

[0m
Compiled Resource:[0m
------------------[0m
# Declared in /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:22:in `from_file'

execute("apt-get-update-build-essentials") do
  action [:nothing]
  retries 0
  retry_delay 2
  command "apt-get update"
  backup 5
  returns 0
  cookbook_name :"build-essential"
  recipe_name "debian"
  not_if { #code block }
end
[0m

[0m
[0m
================================================================================[0m
[31mRecipe Compile Error in /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/default.rb[0m
================================================================================[0m

[0m
Mixlib::ShellOut::CommandTimeout[0m
--------------------------------[0m
execute[apt-get-update-build-essentials] (build-essential::debian line 22) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
Command execeded allowed execution time, killed by TERM signal.
---- Begin output of apt-get update ----
STDOUT: Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Ign http://ppa.launchpad.net trusty InRelease
Err http://ppa.launchpad.net trusty Release.gpg
  Connection failed
Ign http://ppa.launchpad.net trusty Release
Ign http://ppa.launchpad.net trusty/main amd64 Packages/DiffIndex
Ign http://ppa.launchpad.net trusty/main i386 Packages/DiffIndex
STDERR: 
---- End output of apt-get update ----
Ran apt-get update returned [0m

[0m
Cookbook Trace:[0m
---------------[0m
  /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:30:in `from_file'
  /hipchat-scm/chef-repo/cookbooks/build-essential/recipes/default.rb:21:in `from_file'[0m

[0m
Relevant File Content:[0m
----------------------[0m
/hipchat-scm/chef-repo/cookbooks/build-essential/recipes/debian.rb:

 23:    command "apt-get update"
 24:    action :nothing
 25:    # tip: to suppress this running every time, just use the apt cookbook
 26:    not_if do
 27:      ::File.exists?('/var/lib/apt/periodic/update-success-stamp') &&
 28:      ::File.mtime('/var/lib/apt/periodic/update-success-stamp') > Time.now - 86400*2
 29:    end
 30>> end.run_action(:run) if node['build_essential']['compiletime']
 31:  
 32:  %w{
 33:    autoconf
 34:    binutils-doc
 35:    bison
 36:    build-essential
 37:    flex
 38:  }.each do |pkg|
 39:  
[0m

[0m
[2016-11-23T13:28:35+08:00] INFO: Forking chef instance to converge...
[2016-11-23T14:29:32+08:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

診断

  • From server's terminal/command-line, run cat /etc/apt/sources.list to verify if there are other resources in the APT configuration

  • From server's terminal/command-line, run apt-get update; echo $? and check the returned output. The expected output is as follows:

    ~$ sudo dont-blame-hipchat -c "apt-get update; echo $?"
Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en                 
Ign file: hipchat/main Translation-en                
Reading package lists... Done
0

    If the command didn't complete and/or it's trying to reach out to an external URL, proceed with the Resolution section below

原因

APT (Advanced Package Tool) which is a set of tools for managing Debian packages configuration on the server is messed up or there are other resources in the APT configuration. Hipchat Server is not meant to reach out to any external sources during an apt-get operation (apt-get should only hit local files) but as you can see from the stack trace above, the apt-get is reaching out to external URL http://ppa.launchpad.net

ソリューション

  1. Backup Hipchat Server for precautionary measure

  2. From the server's terminal/command-line, run the followings to remove unwanted files from these locations:

    sudo dont-blame-hipchat
apt-get clean
rm -f /etc/apt/sources.list.d/*
rm -rf /var/lib/apt/lists/*

  3. After that, run these commands, to check that the new Chef (configuration manager) .deb files made it:

    dpkg -l | grep hc-
ls -ahl /file_store/archive/pool/main/h/hc-chef-repo/
cat /etc/apt/sources.list
ls -ahl /etc/apt/sources.list.d/

  4. The output should be similar to these:

    ~$ dpkg -l | grep hc-
ii  hc-chef-repo                           2016.11.23.192154-34ee8fd            all          config management for HipChat
ii  hc-client-bundle                       2016.11.09.162543-366c47e            all          HipChat desktop and mobile clients
ii  hc-consul                              2016.11.15.023757-45df12a            all          Consul for HipChat
ii  hc-crowd                               2016.11.01.234254-19fb239            all          Atlassian Crowd for HipChat
ii  hc-cumulus                             2016.09.02.191338-4631a52            all          S3 functionality for HipChat
ii  hc-curler                              2016.11.23.145246-8149c13            all          HipChat Curler
ii  hc-emoticons                           2016.11.23.020131-caa9654            all          HipChat Emoticons
ii  hc-fishtank                            2016.11.23.141140-91c4d48            all          HipChat Services
ii  hc-punjab                              2016.09.06.160718-7c3655d            all          BOSH server
ii  hc-s3-signer                           2016.11.22.143244-02f772b            all          S3 Read request signer for HipChat
ii  hc-scissortail                         2016.11.23.141755-ab3e194            all          HipChat data migration
ii  hc-server-cli                          2016.11.22.183242-ca295ec            all          HipChat Server CLI
ii  hc-web                                 2016.11.23.015636-fe5180c            all          web-based functionality for HipChat
ii  hc-webcore                             2016.11.02.003441-61a7270            all          javascript client for HipChat
admin@coco:~$ ls -ahl /file_store/archive/pool/main/h/hc-chef-repo/
total 412K
drwxr-xr-x  2 root root 4.0K Nov 23 19:29 .
drwxr-xr-x 23 root root 4.0K Nov 23 19:29 ..
-rw-r--r--  1 root root 400K Nov 23 19:29 hc-chef-repo_2016.11.23.192154-34ee8fd_all.deb
admin@coco:~$ cat /etc/apt/sources.list
#
# Ubuntu trusty - Generated by Chef
#

deb [arch=amd64] file:///file_store/archive trusty main
deb [arch=amd64] file:///file_store/archive hipchat main

admin@coco:~$
admin@coco:~$ ls -ahl /etc/apt/sources.list.d/
total 8.0K
drwxr-xr-x 2 root root 4.0K Mar 22  2016 .
drwxr-xr-x 7 root root 4.0K Nov 23 20:33 ..
admin@coco:~$

  5. Next, run these commands to ensure the right keys are in place:

    apt-key add /file_store/archive/archive-hcsinf-pub.gpg
dpkg --configure -a
apt-get update

    (info) If you receive gpg: can't open `/file_store/archive/archive-hcsinf-pub.gpg': No such file or directory when running the apt-key command above, directly download the apt key to your server following these commands:

    cd /file_store/archive
wget https://s3.amazonaws.com/hipchat-server-stable/utils/archive-hcsinf-pub.gpg
apt-key add /file_store/archive/archive-hcsinf-pub.gpg

  6. The output should look like this:

    # apt-key add /file_store/archive/archive-hcsinf-pub.gpg
OK
# dpkg --configure -a
# apt-get update
Get:1 file: trusty InRelease [1781 B]
Get:2 file: hipchat InRelease [1753 B]
Ign file: trusty/main Translation-en
Ign file: hipchat/main Translation-en
Reading package lists... Done

  7. Finally, re-run the upgrade and wait till it completes:

    hipchat upgrade --force-upgrade

  8. You may tail both update.log and chef.log to check on the progress:

    tail -f /var/log/hipchat/update.log /var/log/chef.log

 

最終更新日 2018 年 11 月 2 日

この内容はお役に立ちましたか?

はい
いいえ
この記事についてのフィードバックを送信する

関連コンテンツ

  • 関連コンテンツがありません
Powered by Confluence and Scroll Viewport.