Title
Mobile Commons: RCA - 2024/01/22 - Delay in processing uploaded groups
Information
TITLE: Mobile Commons – Delay in processing uploaded groups – 2024/01/22
DATE, TIME AND DURATION
START TIME: 2024/01/22 - 12:12 ET RESOLUTION TIME: 2024/01/23 – 10:00 ET DURATION: AROUND 0D 21H 48M
OVERVIEW
On Monday, January 22nd around 12:12 ET, the Mobile Commons support team started receiving reports from customers about delays in the backend processing of uploading/populating profiles to groups. This affected the broadcasts that targeted newly created groups during the outage time.
The Mobile Commons engineering team started investigating immediately and found that this was related to locking errors in the analytic database, which caused the data loading tasks to fail. The engineering team started to do manual rerunning of all the failed tasks individually to ensure each was completed successfully to resolve the issue.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: The analytic database tasks failed due to locking issues on the data load table, which led to the suspension of the main data load task.
- REMEDIATION PLAN:
- Short term: Each of the individual failed tasks was rerun manually to ensure each was completed successfully.
- Long term: Additional monitoring and alerting will be added to the data load tasks to ensure similar issues are caught and resolved as soon as possible.
- RCA ISSUE ID: MC-13605
Keywords RCA 2024 MC_RCA raexternalrequest
Solution ID
240202115658620
Last Modified Date
02/02/2024 12:04:18 PM
Renew Date
02/02/2025
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis
Collections
- UMM Self-Service
- UMM Support
Upland RightAnswers Portal - Version 2024R2