Title

Mobile Commons: RCA - 2022/07/22 - MMS outage

Information

TITLE: Mobile Commons - MMS outage – 2022/07/22

DATE, TIME AND DURATION

START TIME: 2022/07/22 – 18:00 ET RESOLUTION TIME: 2022/07/23 – 22:45 ET DURATION: 1D 4H 45M

OVERVIEW

The Upland Mobile Commons engineering team identified an issue where MMS broadcasts were not sending out as they were getting stuck during the message generation process. The investigation showed that this is related to a database (DB) configuration issue where one of the columns reached its max size and blocked the DB from writing new records to this table.

ROOT CAUSE & REMEDIATION

ROOT CAUSE: The ID column for the MMS Attachment table reached its max size and blocked the DB from writing any new records to that table, which resulted in blocking the MMS messaging traffic.
REMEDIATION PLAN:
- Short term: The team immediately started working on changing the field type to allow storing higher values and making sure this change is reflecting/syncing correctly across the system to restore MMS messaging.
- Long term: The cloud team is working on adding DB alerting so we can get notified when a DB column is close to reaching the max size so we can do the necessary adjustments in advance and avoid any failures.
RCA ISSUE ID: [MC-10608]

Keywords RCA RCA_7.22.2022 import document raexternalrequest

Solution ID

220728163308617

Last Modified Date

07/28/2022 04:38:42 PM

Renew Date

07/28/2023

Status

Published

Author

bmagdy@uplandsoftware.com

Taxonomy

Collections

Upland RightAnswers Portal - Version 2024R2