TITLE: Mobile Commons - MMS outage – 2022/07/22

Date, Time and Duration

START TIME: 2022/07/22 – 18:00 ET

RESOLUTION TIME: 2022/07/23 – 22:45 ET

DURATION: 1D 4H 45M

Overview

• The Upland Mobile Commons engineering team identified an issue where MMS broadcasts were not sending out as they were getting stuck during the message generation process. The investigation showed that this is related to a database (DB) configuration issue where one of the columns reached its max size and blocked the DB from writing new records to this table.

Root Cause & remediation

• ROOT CAUSE: The ID column for the MMS Attachment table reached its max size and blocked the DB from writing any new records to that table, which resulted in blocking the MMS messaging traffic.
• REMEDIATION PLAN:

-          Short term: The team immediately started working on changing the field type to allow storing higher values and making sure this change is reflecting/syncing correctly across the system to restore MMS messaging.

-          Long term: The cloud team is working on adding DB alerting so we can get notified when a DB column is close to reaching the max size so we can do the necessary adjustments in advance and avoid any failures.

• RCA ISSUE ID: [MC-10608]