TITLE: Mobile Commons – Delay in processing uploaded groups

Date, Time and Duration

Occurrence Time:

• Monday, 1/22/2024, 12:12 ET - Tuesday, 1/23/2024, 10:00 ET 
• Tuesday, 2/6/2023, 13:15 ET - Tuesday, 2/6/2023, 13:15 ET 

Duration:

• Monday, 1/22/2024: 21 hours, 48 minutes 
• Tuesday, 2/6/2023: 58 minutes

Overview

On Monday, January 22nd around 12:12 ET, the Mobile Commons support team started receiving reports from customers about delays in the backend processing of uploading/populating profiles to groups. This affected the broadcasts that targeted newly created groups during the outage time.  

The Mobile Commons engineering team started investigating immediately and found that this was related to locking errors in the analytic database, which caused the data loading tasks to fail. The engineering team started manually rerunning all the failed tasks individually to ensure each was completed successfully to resolve the issue as an immediate/short-term solution. Additionally, the team started investigating the implementation of a long-term solution to the issue’s root cause.
 

On Tuesday, February 6th, around 12:17 ET, the Mobile Commons support team received new customer reports of similar delays in the backend processing of uploading/populating profiles to groups due to the same root cause. The engineering team followed the same approach of manually re-running the failed tasks which restored normal performance. On February 14th, the team deployed a fix to address the root cause of the data load failures.

Root Cause & remediation

• ROOT CAUSE: Locking issues on the data load table resulted in the failure of analytic database tasks, consequently causing the suspension of the main data load task.
• REMEDIATION PLAN:

-          Short term: Manual rerunning of individual failed tasks to ensure successful completion.

-          Long term: We have deployed a targeted solution to rectify the underlying root cause responsible for the failure of data load tasks. This adjustment is designed to serve as a robust and enduring resolution for the long term.

• RCA ISSUE ID: MC-13665