How to migrate from Hipchat Server to Hipchat Data Center
目的
This article explains how to use the Hipchat Data Center migration tool. This tool exports data from Hipchat Server version 2.2.3 or later, and imports it into Hipchat Data Center.
This tool is separate from, and incompatible with current export feature available in Hipchat Server. Data exported using the hipchat export
CLI cannot be migrated using this tool.
移行される情報
Migrated | Not migrated |
---|---|
|
|
プロセス
Check your Hipchat Server version
This tool exports data from Hipchat Server version 2.2.3 or later only. You may need to upgrade your Hipchat Server instance to proceed.
Check your Hipchat Server utilization
Before you start the migration process, you should make sure you know how many files, and how many messages you have in your existing Hipchat Server instance. The migration tool requires enough memory on the Hipchat Server instance that you're running it on to process thousands of messages and gigabytes of files.
In general:
- For every 1000 messages, the migration tool will consume an additional 2.6 MB of memory.
- For every 1GB of files, the migration tool will consume an additional 500 KB of memory.
Once you know your file store size and how many messages you have, you can calculate the additional memory requirements. You might need to provision your Hipchat Server instance with more memory to complete the migration tool run. Once the tool has run, you can reduce the instance's memory back to the original amount, if needed.
(Note that this tool runs an all-or-nothing export. If you continue to use your Hipchat Server instance after running the export phase of the migration, you will naturally have some chat history that is not included in the export.)
Install Hipchat Data Center
- Deploy Hipchat Data Center with at least one node.
- Use the Setup Wizard to configure the FQDN, License, etc.
- Make sure no users will be using the system during the import process.
- Configure the email server on the destination server by going to System > Email server in the admin UI.
(Unless your organization uses an external authentication source, all users will receive a password reset after migration, so email is important!)
Install the migration utility
Download the migration utility to your local computer: https://s3.amazonaws.com/hipchat-server-stable/utils/migrate/hc-migrate
v3.3.2, last updated May 24th 2019. SHA512 sum:
you may need to scroll --> 0f37ecb30774ac903f46bde055d24b2b6c320dc263b0928f46187ea93e5ae2f0d72df9efb3569a1f8f7837fb59b0080500359a4edfc2a631d0cbccc7d6fb3b90
Copy the migration utility to the source Hipchat Server.
scp -i /path/to/sshkey.pem hc-migrate admin@<hcserver.example.com>:/home/admin/hc-migrate
Copy the migration utility to a node on the destination Hipchat Data Center instance.
scp -i /path/to/sshkey.pem hc-migrate admin@<hcdcnode.example.com>:/home/admin/hc-migrate
Depending on your environment, you may need to copy the migration utility to your data center's jumpbox (or management portal) then copy it to the node.
Export from Hipchat Server
Reminder! This tool exports data from Hipchat Server version 2.2.3 or later only. You may need to upgrade to a compatible version of Hipchat Server to proceed.
Connect to the Hipchat Server command-line interface using a terminal or SSH
ssh -i /path/to/sshkey.pem admin@<hcserver.example.com>
Escalate your user permissions to root
sudo dont-blame-hipchat
Switch to the directory containing the migration utility
cd /home/admin/
Update the permissions for the migration utility so it can be run, verify the checksum and the version.
chmod +x /home/admin/hc-migrate sha512sum hc-migrate /home/admin/hc-migrate -version Hipchat Migration Utility version 3.1.4 (2018-02-01)
Retrieve the password to the source Hipchat Server MariaDB database. You'll use this (without the quotation marks in the next step.
cat /hipchat/config/site.json | jq -r .databases.hipchat.pass
Copy, edit, and run the following example command to start the export.
By default, the export artifact is saved in the directory from which you run the command. You can use the the
--export
flag to specify another location./home/admin/hc-migrate --export "exportFileLocation.tar.gz.aes" --aes_password "passwordToExportFile" \ --db_password "databasePassword" --verbose |& tee desiredLogFile
A sample command would look something like this:
/home/admin/hc-migrate --export "/home/admin/20171115_migrate.tar.gz.aes" --aes_password "hipchat" --db_password "reallysecurepassword" --verbose |& tee /home/admin/newmigrate_export.log
When the export is complete the logs will stop in the terminal and the command will exit. The end of the log output for a successful export looks like this:
DEBUG 2017/11/15 01:59:04 Writing rooms/711/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/712/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/713/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/714/history.json DEBUG 2017/11/15 01:59:04 Writing rooms/715/history.json DEBUG 2017/11/15 01:59:04 ======== *** Export completed successfully! *** ========
- Copy the resulting export artifact to a workstation or other system with access to the Hipchat Data Center node
Import into Hipchat Data Center
- Copy the export artifact to a Hipchat Data Center node
Connect to the Hipchat Data Center node command-line interface using a terminal or SSH
ssh -i /path/to/sshkey.pem admin@<hcdcnode.example.com>
Escalate your user permissions to root
sudo dont-blame-hipchat
Switch to the directory containing the migration utility
cd /home/admin/
Update the permissions for the migration utility so it can be run, verify the checksum and the version.
chmod +x /home/admin/hc-migrate sha512sum hc-migrate /home/admin/hc-migrate -version Hipchat Migration Utility version 3.1.4 (2018-02-01)
You'll need your Redis and Postgres connection details for the next steps. If you don't already have them, run the following commands get them.
Postgres connection detailsjq -r '.databases.hipchat_postgres' /hipchat/config/site.json
Redis connection detailspwd=$(jq -r '.databases.hipchat_postgres.pass' /hipchat/config/site.json) usr=$(jq -r '.databases.hipchat_postgres.user' /hipchat/config/site.json) host=$(jq -r '.databases.hipchat_postgres.servers[0]' /hipchat/config/site.json | cut -d: -f1) schema=$(jq -r '.databases.hipchat_postgres.schema' /hipchat/config/site.json) PGPASSWORD=$pwd psql -h $host -U $usr -d $schema -t -c "SELECT * FROM configurations WHERE key LIKE 'redis%';"
Run the command below to clear the Redis cache.
This will remove all key:value pairs from the Redis environment specified, regardless of whether or not they are associated with Hipchat.
redis-cli -h <RedisHost> -a <Password> flushdb
Sample Outputroot@ip-10-10-10-12:/home/admin# jq -r '.databases.hipchat_postgres' /hipchat/config/site.json { "user": "hcdc", "servers": [ "10.10.10.13:5432" ], "schema": "hipchat_postgres", "pass": "super-secret-pw" } root@ip-10-10-10-12:/home/admin# pwd=$(jq -r '.databases.hipchat_postgres.pass' /hipchat/config/site.json) root@ip-10-10-10-12:/home/admin# usr=$(jq -r '.databases.hipchat_postgres.user' /hipchat/config/site.json) root@ip-10-10-10-12:/home/admin# host=$(jq -r '.databases.hipchat_postgres.servers[0]' /hipchat/config/site.json | cut -d: -f1) root@ip-10-10-10-12:/home/admin# schema=$(jq -r '.databases.hipchat_postgres.schema' /hipchat/config/site.json) root@ip-10-10-10-12:/home/admin# PGPASSWORD=$pwd psql -h $host -U $usr -d $schema -t -c "SELECT * FROM configurations WHERE key LIKE 'redis%';" redishostname | 10.10.10.13 redisport | 6379 redispass | supersecretredispassword root@ip-10-10-10-12:/home/admin# redis-cli -h 10.10.10.13 -a supersecretredispassword flushdb OK
Copy, edit, and run the example command below to start the import process.
Running the import with the
--forceclean
flag will delete all existing user accounts in the database (including the owner account created during the setup wizard)./home/admin/hc-migrate --import "exportFileLocation.tar.gz.aes" --aes_password "passwordToExportFile" \ --postgres_url "dbname=hipchat_postgres host=postgresIpAddressorDomain user=postgresUsername password=postgresPassword port=postgresPort sslmode=disable" \ --redis_url "redisIpOrHostname:RedisPort" --redis_password "redisPassword" \ --forceclean --verbose |& tee <desiredLogFile>
A sample command would look something like this:
/home/admin/hc-migrate --import "hcs_export_05-12-2017.tar.gz.aes" --aes_password "asdf9324lkJasdf" \ --postgres_url "dbname=hipchat_postgres host=10.10.10.7 user=hcdc password=supersecretpostgres port=5432 sslmode=disable" \ --redis_url "10.10.10.6" --redis_password "supersecretredis" \ --forceclean --verbose |& tee <desiredLogFile>
When the import finishes, the terminal window stops displaying new log updates, and the command exits. Look for the fancy ASCII art at the end of the log output which indicates a successful export:
DEBUG 2017/11/15 02:41:19 Counted 3 messages in muc-2017.11 DEBUG 2017/11/15 02:41:19 == Extracting and processing rooms/714/history.json DEBUG 2017/11/15 02:41:19 Counted 3 messages in muc-2017.11 DEBUG 2017/11/15 02:41:19 == Extracting and processing rooms/715/history.json DEBUG 2017/11/15 02:41:19 Counted 3 messages in muc-2017.11 DEBUG 2017/11/15 02:41:19 total message count 16688 DEBUG 2017/11/15 02:41:19 Done processing the source DEBUG 2017/11/16 21:38:27 ======== *** Import completed successfully! *** ========
Run the command below to restart Hipchat. Run this command on each node in the HCDC cluster.
hup
- Request a password reset for each user that will be logging in to HCDC using the internal authentication system at https://example.hcdc.com/forgot_password
(If your deployment uses an external user directory, such as LDAP or SAML, you don't need to do this.) - Complete any necessary set up for Hipchat Data Center and start chatting.
Performance profiling for the migration tools
If your migration job is not completing successfully, or is having other issues, you might want to collect performance data on your Hipchat deployment to better understand what's going on. Before you begin either the the export or the import process, you can open a new terminal window with a new SSH session so you can monitor and log performance data while other operations are taking place.
Open a new terminal or SSH session into the deployment you'll be working on.
Collect a baseline of system utilization (wait ~1 minute for this to finish).
Replace "impbase" in these commands with a filename that indicates to you if the file is from an import or an export process.date > /home/admin/impbase_vmstat.log && vmstat -w 3 20 | tee -a /home/admin/impbase_vmstat.log
Start collecting system utilization (this will run until cancelled).
Replace "import" in these commands with a filename that indicates to you if the file is from an import or an export process.
date > /home/admin/import_vmstat.log && vmstat -w 3 | tee -a /home/admin/import_vmstat.log
- Run the export or import task in the other terminal or SSH session. When it completes, return to the profiling session.
Press Ctrl + C to cancel the
vmstat
run.Append a stop stamp:
date >> /home/admin/import_vmstat.log
Expected Errors
"AES password incorrect, cannot decrypt archive. Please try again" |
|
"Cannot begin import, destination database contains user data. Use --forceclean to overwrite" |
|
Migration tool flag reference
グローバル
Flag | What does it do | 既定 |
---|---|---|
--version, -v | Output the current version of the migrate tool | false |
--verbose; -j | Enable verbose logging | false |
--help -h | Shows help message | false |
インポート
Flag | What does it do | 既定 |
---|---|---|
--import -i | The location to the exported artifact that will be imported | なし |
--aes_password | The password to use to decrypt the export artifact | なし |
--forceclean | Clean tables before starting the import. Ensures the database is in a good state for importing | false |
--hcs | Import to HCS, default is to HCDC | false |
エクスポート
Flag | What does it do | 既定 |
---|---|---|
--export -e | The location of where to save the exported artifact | なし |
--aes_password | The password to use to encrypt the export artifact | なし |
--hcdc | Export from HCDC, default is from HCS | false |
--start_date | The earliest date to fetch history in the RFC3339 format If set, messages before this date will be skipped. | 空 |
--end_date | The latest date to fetch history in the RFC3339 format If set, messages after this date will be skipped. | 空 |
--skip_attachments | Skip attachments in the export | false |
RFC339 format yyyy-mm-ddThh:mm:ss+00:00
, for example: 2018-05-01T00:00:00+00:00
Data Access
Flag | What does it do | 既定 |
---|---|---|
--elasticsearch | The HCS Elasticsearch endpoint for export | http://localhost:9200/ |
--db_user | The HCS MariaDB/MySQL DB username | root |
--db_password | The HCS MariaDB/MySQL DB password | 空 |
--postgres_url | The HCDC Postgres database to connect to | postgres://localhost |
--redis_url | Redis instance to connect to | localhost:6379 |
--redis_password | Redis instance password | 空 |
--redis_stats_url | Redis Stats instance to connect to | localhost:6380 |
--redis_stats_password | Redis Stats instance password | 空 |