Integrations no longer send notifications to rooms after server upgrade
プラットフォームについて: Server および Data Center のみ。この記事は、Server および Data Center プラットフォームのアトラシアン製品にのみ適用されます。
Support for Server* products ended on February 15th 2024. If you are running a Server product, you can visit the Atlassian Server end of support announcement to review your migration options.
*Fisheye および Crucible は除く
This is for an outdated version of Hipchat Server
This article applies to a version of Hipchat Server which will be deprecated soon. After that period the version will no longer be supported.
You should upgrade to a more recent version of Hipchat Server as soon as you can to take advantage of new features, and security and bug fixes.
問題
Integrations are no longer sending notifications to rooms. Some examples are:
- Jira no longer send notifications to Hipchat rooms
- Confluence no longer send notifications to Hipchat rooms
The following appears in the /home/admin/logs/nginx.log:
2016-03-17T15:07:01.672700+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] =====> New API request: #1522, uri=/room/3/notification, method=POST, from=127.0.0.1, job_handle=none
2016-03-17T15:07:01.672888+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Fetching an oauth session for token XXXXXXXXUsu7SWiUjPvZpaJdmt1DYNOtxwAvcXC2
2016-03-17T15:07:01.673066+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query: SELECT * FROM oauth_sessions WHERE access_token=%(access_token)s, args={'access_token': 'XXXXXXXXUsu7SWiUjPvZpaJdmt1DYNOtxwAvcXC2'}
2016-03-17T15:07:01.673986+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query completed in time=1ms
2016-03-17T15:07:01.674263+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Authenticated session 8181 by access token XXXXXXXXUsu7SWiUjPvZpaJdmt1DYNOtxwAvcXC2
2016-03-17T15:07:01.675277+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Authenticated client c9b73251-34ef-4ef9-b454-87eeea5e29e6
2016-03-17T15:07:01.677322+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query: SELECT group_id, name FROM rooms WHERE group_id=%(group_id)s AND id = %(room_id)s AND !is_deleted , args={'pretty_name': u'3', 'group_id': 1, 'room_id': 3}
2016-03-17T15:07:01.677910+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] SQL query completed in time=1ms
2016-03-17T15:07:01.679008+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Sending system message to rid-1_dropcam_devops, data={'color': 'yellow', 'from_nick': 'Frodo', 'notify': False, 'message_format': u'text'}
2016-03-17T15:07:01.679507+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] ERROR: Error calling broadcast_room_system_message: Service Unavailable, try back later
2016-03-17T15:07:01.679666+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] ERROR: Sending API error: 503 - Service Unavailable, try back later
2016-03-17T15:07:01.680065+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] Traceback (most recent call last):
2016-03-17T15:07:01.680107+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
2016-03-17T15:07:01.680147+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] result = g.send(result)
2016-03-17T15:07:01.680186+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/libs/remote_proxy.py", line 158, in proxy
2016-03-17T15:07:01.680224+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] resp = yield self.call_remote_server(func_name, body)
2016-03-17T15:07:01.680262+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/dace/call.py", line 194, in __call__
2016-03-17T15:07:01.680299+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] d = self._run(*args, **kwargs)
2016-03-17T15:07:01.680337+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1237, in unwindGenerator
2016-03-17T15:07:01.680373+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] return _inlineCallbacks(None, gen, Deferred())
2016-03-17T15:07:01.680409+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] --- <exception caught here> ---
2016-03-17T15:07:01.680446+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
2016-03-17T15:07:01.680482+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] result = g.send(result)
2016-03-17T15:07:01.680519+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/dace/_decorator.py", line 97, in call
2016-03-17T15:07:01.680555+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] d = yield func(*args, **kwargs)
2016-03-17T15:07:01.680592+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/dace/_decorator.py", line 129, in call
2016-03-17T15:07:01.680629+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] scheme, netloc = load_balancer()
2016-03-17T15:07:01.680666+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] File "/hipchat-scm/coral/vendor/virtualenv/local/lib/python2.7/site-packages/txhipchat/tetra/app_load_balancer.py", line 41, in __call__
2016-03-17T15:07:01.680701+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] raise ServiceUnavailableError("No tetra hosts found")
2016-03-17T15:07:01.680738+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] txhipchat.error.ServiceUnavailableError: Service Unavailable, try back later
2016-03-17T15:07:01.680813+00:00 hipchat coral-1: [/room/3/notification#REQ0G9KpG] WARN: =====> API request failed: #1522, in-progress=0, code=503
2016-03-17T15:07:03.468886+00:00 hipchat nginx-access: 54.165.71.43 - - [17/Mar/2016:15:07:01 +0000] "POST /v2/room/3/notification HTTP/1.1" 503 136 "-" "python-requests/2.9.1" "-" -From the log snippet above, you can see there's an error sending the notification and the API request failed. It occurs when hitting the '/room/<ID>/notification' REST endpoint and it failed when trying to connect to a tetra-app service which responsible for a big piece of actually sending the notification.
診断
Re-configure the integrations to isolate caching issue.
原因
tetra-app cache entries in the server are broken.
ソリューション
Restart the tetra-app and related services with the following command:
hipchat service --restart
The above command should re-register cached 'tetra-app' entries we're looking for, allowing the process to send notifications to complete successfully.
This will cause a brief service outage, so it is advisable to notify your team upfront or set a small maintenance window to avoid service disruption.
If there's still an issue, please contact Hipchat Server Support here.