Title
Mobile Commons: RCA - 2024/02/06 - Delay in processing uploaded groups
Information
TITLE: Mobile Commons – Delay in processing uploaded groups
Occurrence Time:
DATE, TIME AND DURATION
- Monday, 1/22/2024, 12:12 ET - Tuesday, 1/23/2024, 10:00 ET
- Tuesday, 2/6/2023, 13:15 ET - Tuesday, 2/6/2023, 13:15 ET
Duration:
- Monday, 1/22/2024: 21 hours, 48 minutes
- Tuesday, 2/6/2023: 58 minutes
OVERVIEW
On Monday, January 22nd around 12:12 ET, the Mobile Commons support team started
receiving reports from customers about delays in the backend processing of uploading/populating profiles to groups. This affected the broadcasts that targeted newly created groups during the outage time.
The Mobile Commons engineering team started investigating immediately and found that this was related to locking errors in the analytic database, which caused the data loading tasks to fail. The engineering team started manually rerunning all the failed tasks individually to ensure each was completed successfully to resolve the issue as an immediate/short-term solution.
Additionally, the team started investigating the implementation of a long-term solution to the issue’s root cause.
On Tuesday, February 6th, around 12:17 ET, the Mobile Commons support team received new customer reports of similar delays in the backend processing of uploading/populating profiles to groups due to the same root cause. The engineering team followed the same approach of manually re-running the failed tasks which restored normal performance. On February 14th, the team deployed a fix to address the root cause of the data load failures.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: Locking issues on the data load table resulted in the failure of analytic
database tasks, consequently causing the suspension of the main data load task.
REMEDIATION PLAN:
- Short term: Manual rerunning of individual failed tasks to ensure successful completion.
- Long term: We have deployed a targeted solution to rectify the underlying root cause responsible for the failure of data load tasks. This adjustment is designed to serve as a robust and enduring resolution for the long term.
- RCA ISSUE ID: MC-13665
Keywords RCA 2024 MC_RCA raexternalrequest
Solution ID
240227140523617
Last Modified Date
02/27/2024 02:08:32 PM
Renew Date
03/01/2026
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis
Collections
- UMM Self-Service
- UMM Support
Upland RightAnswers Portal - Version 2024R2