TITLE: Mobile Commons – Outgoing messages stuck as "Queued"

Date, Time and Duration

START TIME: 2024/05/15 – 23:12 ET

RESOLUTION TIME: 2024/05/16 – 20:00 ET

Duration: 20 hours, 48 minutes

Overview

On Thursday, May 16th, 2024, around 3:03 PM EDT, reports came in about messages being stuck in a "queued" status, affecting several accounts and causing delays.

Upon receiving reports, the Mobile Commons support team immediately launched an investigation to determine the scope of the issue and identify any additional affected accounts. Simultaneously, efforts were directed towards isolating the root cause of the problem and implementing measures to restore normal message delivery functionality. A maintenance window following the MC May release exposed disruptions stemming from an upgrade to the system's caching storage service. These disruptions predominantly impacted certain worker processes, requiring manual restarts to restore functionality.

Manual restart to the affected application workers resolved the issue, and things returned to normal by 8:00 PM EDT on May 16th.

Root Cause & remediation

• ROOT CAUSE: Upgrade to the caching storage services caused the disruption due to some application workers not restarting on their own.
• REMEDIATION PLAN:

-          Short term: Manual restart of affected application workers.

-          Long term: Implementing proactive monitoring for the caching worker processes.

• RCA ISSUE ID: MC-14149