Title
Mobile Commons: Outgoing messages stuck as "Queued" - 5/16/2024
Information
TITLE: Mobile Commons – Outgoing messages stuck as "Queued"
DATE, TIME AND DURATION
START TIME: 2024/05/15 – 23:12 ET
RESOLUTION TIME: 2024/05/16 – 20:00 ET
Duration: 20 hours, 48 minutes
OVERVIEW
On Thursday, May 16th, 2024, around 3:03 PM EDT, reports came in about messages being
stuck in a "queued" status, affecting several accounts and causing delays.
Upon receiving reports, the Mobile Commons support team immediately launched an investigation to determine the scope of the issue and identify any additional affected accounts. Simultaneously, efforts were directed towards isolating the root cause of the problem and implementing measures to restore normal message delivery functionality. A maintenance window following the MC May release exposed disruptions stemming from an upgrade to the system's caching storage service. These disruptions predominantly impacted certain worker processes, requiring manual restarts to restore functionality.
Manual restart to the affected application workers resolved the issue, and things returned to normal by 8:00 PM EDT on May 16th.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: Upgrade to the caching storage services caused the disruption due to
some application workers not restarting on their own.
- REMEDIATION PLAN:
- Short term: Manual restart of affected application workers.
- Long term: Implementing proactive monitoring for the caching worker processes.
- RCA ISSUE ID: MC-14149
Keywords RCA raexternalrequest
Solution ID
240604204719127
Last Modified Date
06/04/2024 08:50:01 PM
Renew Date
06/04/2025
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis
Collections
- UMM Self-Service
- UMM Support
Upland RightAnswers Portal - Version 2024R2