Title
Mobile Commons: RCA - 2023/5/18 - Broadcast generation outage
Information
TITLE: Mobile Commons – Broadcasts generation outage – 5/18/2023
DATE, TIME AND DURATION
START TIME: 2023/05/18 – 15:25 ET
RESOLUTION TIME: 2023/05/18 – 17:05 ET
DURATION: 0D 1H 40M
OVERVIEW
On Thursday 5/18/2023 at approx. 15:25 ET, the MC support team received reports of scheduled
broadcasts that failed to send on time. Our engineering team began investigating immediately and discovered an edge scenario in which broadcast generation processes would freeze and not process the scheduled broadcasts. At 17:05 ET, the team was able to get the stuck broadcasts to send normally.
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
The root cause was found to be related to an edge scenario in which the generation process for broadcasts and the parts/time-zone broadcasts created a race condition. This condition caused all the generation processes to attempt to work on the same broadcast over and over, causing the generation process to get stuck and affect all the broadcasts in the queue.
REMEDIATION PLAN:
Short Term:
- The team was able to work immediately to process the affected scheduled broadcasts and push them to send normally.
- Monitor the broadcast generation queue to ensure normal operations.
Long Term:
The team pushed out a code change to address this edge scenario and better handle the generation process for time-zone broadcasts.
- RCA ISSUE ID: [MC-12481]
Keywords RCA broadcast_generation outage raexternalrequest
Solution ID
230615105626093
Last Modified Date
01/24/2024 10:27:46 AM
Renew Date
02/01/2026
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis > 2023
Collections
- UMM Self-Service
- UMM Support
Upland RightAnswers Portal - Version 2024R2