TITLE: Mobile Commons – Broadcasts generation outage – 5/18/2023

Date, Time and Duration

START TIME: 2023/05/18 – 15:25 ET

RESOLUTION TIME: 2023/05/18 – 17:05 ET

DURATION: 0D 1H 40M

Overview

On Thursday 5/18/2023 at approx. 15:25 ET, the MC support team received reports of scheduled broadcasts that failed to send on time. Our engineering team began investigating immediately and discovered an edge scenario in which broadcast generation processes would freeze and not process the scheduled broadcasts. At 17:05 ET, the team was able to get the stuck broadcasts to send normally.

Root Cause & remediation

• ROOT CAUSE:

The root cause was found to be related to an edge scenario in which the generation process for broadcasts and the parts/time-zone broadcasts created a race condition. This condition caused all the generation processes to attempt to work on the same broadcast over and over, causing the generation process to get stuck and affect all the broadcasts in the queue.

• REMEDIATION PLAN:

Short Term:

1-      The team was able to work immediately to process the affected scheduled broadcasts and push them to send normally.

2-      Monitor the broadcast generation queue to ensure normal operations.

Long Term:

The team pushed out a code change to address this edge scenario and better handle the generation process for time-zone broadcasts.

• RCA ISSUE ID: [MC-12481]