Mobile Commons: RCA - 2021/02/08 - Truth Initiative Concatenated MT Outage
Information
| Title: | Truth Initiative Concatenated MT Outage – 02/05/2021 | |||||
| Incident Number: | MC-20210205 | Ticket/Request Number: | #MC-8227 | Date: | 08 Feb 2021 | |
Description
The purpose of this Root Cause Analysis (RCA) is to provide the factors that caused concatenated MTs for Truth on shortcode 88709 to not be sent from the Mobile Commons platform on February 4-5, 2021. | ||||||
ISSUE DURATION
The issue was first noticed at 3:00 PM EST on Thursday, February 4th and was resolved at 4:45 PM EST | ||||||
Detail | ||||||
|
Mobile Commons support notified the engineering team at 3:00 PM on Thursday, February 4th that reports of some outgoing MTs failing to be delivered to devices had been received. The initial examples provided were referencing only a small number of messages on shared shortcodes and additional logs showed that this was affecting only a small percentage of outgoing texts. Truth Initiative alerted Mobile Commons at 10 AM on Friday with evidence of similar failures and provided more examples for the Mobile Commons engineering team to review which led to identifying the root cause. The solution found was to roll back the scheduled release that had been deployed on Wednesday night, which was done at 4:45 PM EST, and resolved the issue. Additional details on the specific failed messages have been included in this document. | ||||||
|
| ||||||
Root cause
A change was mistakenly introduced in the code that was deployed on the evening of February 3rd that affected some outgoing MT messages. The affected messages were related only to multi-part outbound messages triggered by either keyword, webform, API or MMS broadcast failover. Outbound multi-part messages as part of standard broadcasts were not affected by this issue.
| ||||||
corrective action | ||||||
|
| ||||||
The platform release was rolled back to the previous version to restore functionality.
Keywords
RCA RCA_2021.02.08 Document Upload Import raexternalrequest