TITLE: Mobile Commons – Delay in processing uploaded groups – 2024/01/22

Date, Time and Duration

START TIME: 2024/01/22 - 12:12 ET

RESOLUTION TIME: 2024/01/23 – 10:00 ET

DURATION: AROUND 0D 21H 48M

Overview

On Monday, January 22nd around 12:12 ET, the Mobile Commons support team started receiving reports from customers about delays in the backend processing of uploading/populating profiles to groups. This affected the broadcasts that targeted newly created groups during the outage time.

The Mobile Commons engineering team started investigating immediately and found that this was related to locking errors in the analytic database, which caused the data loading tasks to fail. The engineering team started to do manual rerunning of all the failed tasks individually to ensure each was completed successfully to resolve the issue.

Root Cause & remediation

• ROOT CAUSE: The analytic database tasks failed due to locking issues on the data load table, which led to the suspension of the main data load task.
• REMEDIATION PLAN:

-          Short term: Each of the individual failed tasks was rerun manually to ensure each was completed successfully.

-          Long term: Additional monitoring and alerting will be added to the data load tasks to ensure similar issues are caught and resolved as soon as possible.

• RCA ISSUE ID: MC-13605