Title
Mobile Commons: RCA - 2021/03/11 - Truth Initiative MT Outage
Information
Title: Truth Initiative MT Outage – 03/11/2021
Inciden | MC-20210311 | Ticket/Request #MC-8315 Date: 12 Mar 2021 |
t | Number: | |
Numbe | ||
r: |
DESCRIPTION
The purpose of this Root Cause Analysis (RCA) is to provide the factors that caused outbound MTs for Truth on shortcode 88709 to not be sent from the Mobile Commons platform on March 11, 2021.
ISSUE DURATION
The issue began at 1:45 PM EST on Thursday, March 11th and was resolved at 10:30 PM EST.
DETAIL
The Mobile Commons engineering team was notified at 10:30 PM ET on Thursday, March 11th that opt-in paths were failing to respond and MTs were not being sent out from the platform for Truth Initiative’s short code, 88709. The issue was quickly identified as a configuration issue and was corrected immediately, and traffic was restored. Expected responses were not queued and needed to be manually restored and re-sent.
ROOT CAUSE
This short code underwent some planned maintenance during the day on March 11th beginning at 1:00PM ET. During testing of the changes to the configuration of this code it was not successful so the configuration was rolled back. Unfortunately, the roll-back process was not verified for this code which led to the outgoing messages failing until Upland was notified.
CORRECTIVE ACTION
The configuration of the short code was corrected to restore traffic. Previously undelivered messages were identified, requeued and sent during hours agreed upon with the customer which were completely sent by Monday.
Keywords RCA Truth Import Document RCA_2021.03.11 raexternalrequest
Solution ID
220518102539400
Last Modified Date
05/18/2022 10:31:36 AM
Renew Date
05/18/2023
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis
Collections
- UMM Support
Upland RightAnswers Portal - Version 2024R2