Title
Mobile Commons: HSUS Stuck broadcast 2024/5/21
Information
TITLE: Mobile Commons – HSUS broadcast stuck at 31%
DATE, TIME AND DURATION
START TIME: 2024/05/15 – 23:00 ET
RESOLUTION TIME: 2024/05/29 – 16:00 ET
Duration: 13 days, 17 hours
OVERVIEW
On Tuesday May 21st, 2024, HSUS scheduled a broadcast to go out at 9:15 PM. The broadcast
started sending on time and later got stuck at 31%.
The Mobile Commons engineering team started investigating and tracking down the root cause. It was found to be related to the new broadcast generation technology introduced in May release, where there is an issue causing some broadcasts to get stuck during the automated scaling process of broadcast generators.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: An issue that caused the broadcast processes to hang when the
autoscaling for the broadcast generators triggers.
- REMEDIATION PLAN:
- The team adjusted the timeout of the broadcast queries, so they can resume and complete during similar scenarios.
- Added a protection to disallow the automatic down scaling of active broadcast generators. Which would help avoid similar issues with running broadcasts.
- RCA ISSUE ID: MC-14197
Keywords RCA RCAs raexternalrequest
Solution ID
240604181137767
Last Modified Date
06/04/2024 06:16:06 PM
Renew Date
06/04/2025
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis
Collections
- UMM Support
Upland RightAnswers Portal - Version 2024R2