TITLE: Mobile Commons – HSUS broadcast stuck at 31%

Date, Time and Duration

START TIME: 2024/05/15 – 23:00 ET

RESOLUTION TIME: 2024/05/29 – 16:00 ET

Duration: 13 days, 17 hours

Overview

On Tuesday May 21st, 2024, HSUS scheduled a broadcast to go out at 9:15 PM. The broadcast started sending on time and later got stuck at 31%.

The Mobile Commons engineering team started investigating and tracking down the root cause. It was found to be related to the new broadcast generation technology introduced in May release, where there is an issue causing some broadcasts to get stuck during the automated scaling process of broadcast generators.

Root Cause & remediation

• ROOT CAUSE: An issue that caused the broadcast processes to hang when the autoscaling for the broadcast generators triggers.
• REMEDIATION PLAN:

-          The team adjusted the timeout of the broadcast queries, so they can resume and complete during similar scenarios.

-          Added a protection to disallow the automatic down scaling of active broadcast generators. Which would help avoid similar issues with running broadcasts.

• RCA ISSUE ID: MC-14197