TITLE: Mobile Commons – Broadcasts not sending – 2023/03/20 & 2023/03/21

Date, Time and Duration

Occurrence Time

• Monday, 3/20/2023 12:34 - 13:30 ET
• Tuesday, 3/21/2023 12:57 - 14:52 ET

Duration:

• Monday, 3/20/2023: 56 Minutes
• Tuesday, 3/21/2023: 55 Minutes

Overview

On Monday, 2023/03/20 at 12:33, the Mobile Commons support team received reports of broadcasts that didn’t start the send process. The investigation by our engineering team identified the issue to be related to an edge scenario with the broadcast generation process that caused some of the broadcasts not to start sending correctly. The team pushed out a minor code change as a hotfix that addressed the root cause.

On 2023/03/21 around 12:57 PM ET, the MC support team received additional incidents of broadcasts that didn’t start the send process. The engineering team started investigating the issue immediately and it was found that the new code introduced as a fix on 2023/03/20 was not picked up correctly by some of the broadcast sub-generators. Our cloud team restarted all the generators which forced them to pick up the new code and fixed the issue immediately.

Root Cause & remediation

• ROOT CAUSE:

-          3/20/2023: An edge case with the broadcast generation process that caused some of the broadcasts to not start correctly.

-          3/21/2023: Some of the broadcast sub- generators not picking up the new code fix pushed out on 3/20/2023.

• REMEDIATION PLAN:

-          A code change was introduced to address the edge case that was causing some broadcasts to not start the send process.

-          Immediately restarted all the affected generators to pick up the new code. Once the restart process was completed, all the affected broadcasts started generating normally and things have been working smoothly since then.

-          Long Term: The team has reviewed the process of applying hotfixes and added additional checks for further confirmation and eliminating similar issues.

• RCA ISSUE ID: [MC-12299]