Mobile Commons: RCA - 2021/03/11 - Truth Initiative MT Outage
Information
| Title: | Truth Initiative MT Outage – 03/11/2021 | |||||
| Incident Number: | MC-20210311 | Ticket/Request Number: | #MC-8315 | Date: | 12 Mar 2021 | |
Description
The purpose of this Root Cause Analysis (RCA) is to provide the factors that caused outbound MTs for Truth on shortcode 88709 to not be sent from the Mobile Commons platform on March 11, 2021. | ||||||
ISSUE DURATION
The issue began at 1:45 PM EST on Thursday, March 11th and was resolved at 10:30 PM EST. | ||||||
Detail | ||||||
|
The Mobile Commons engineering team was notified at 10:30 PM ET on Thursday, March 11th that opt-in paths were failing to respond and MTs were not being sent out from the platform for Truth Initiative’s short code, 88709. The issue was quickly identified as a configuration issue and was corrected immediately, and traffic was restored. Expected responses were not queued and needed to be manually restored and re-sent. | ||||||
|
| ||||||
Root cause
This short code underwent some planned maintenance during the day on March 11th beginning at 1:00PM ET. During testing of the changes to the configuration of this code it was not successful so the configuration was rolled back. Unfortunately, the roll-back process was not verified for this code which led to the outgoing messages failing until Upland was notified.
| ||||||
corrective action | ||||||
|
| ||||||
The configuration of the short code was corrected to restore traffic. Previously undelivered messages were identified, requeued and sent during hours agreed upon with the customer which were completely sent by Monday.
Keywords
RCA Truth Import Document RCA_2021.03.11 raexternalrequest