Skip to main content

Mobile Commons: RCA - 2021/02/08 - Truth Initiative Concatenated MT Outage

MC Admin

Title

Mobile Commons: RCA - 2021/02/08 - Truth Initiative Concatenated MT Outage

Information

Title: Truth Initiative Concatenated MT Outage – 02/05/2021

Inciden

MC-20210205

Ticket/Request #MC-8227 Date: 08 Feb 2021

t

Number:

Numbe

r:

DESCRIPTION

The purpose of this Root Cause Analysis (RCA) is to provide the factors that caused concatenated MTs for Truth on shortcode 88709 to not be sent from the Mobile Commons platform on February 4-5, 2021.

ISSUE DURATION

The issue was first noticed at 3:00 PM EST on Thursday, February 4th and was resolved at 4:45 PM EST

DETAIL

Mobile Commons support notified the engineering team at 3:00 PM on Thursday, February 4th that reports of some outgoing MTs failing to be delivered to devices had been received. The initial examples provided were referencing only a small number of messages on shared shortcodes and additional logs showed that this was affecting only a small percentage of outgoing texts. Truth Initiative alerted Mobile Commons at 10 AM on Friday with evidence of similar failures and provided more examples for the Mobile Commons engineering team to review which led to identifying the root cause. The solution found was to roll back the scheduled release that had been deployed on Wednesday night, which was done at 4:45 PM EST, and resolved the issue. Additional details on the specific failed messages have been included in this document.

ROOT CAUSE

A change was mistakenly introduced in the code that was deployed on the evening of February 3rd that affected some outgoing MT messages. The affected messages were related only to multi-part outbound messages triggered by either keyword, webform, API or MMS broadcast failover. Outbound multi-part messages as part of standard broadcasts were not affected by this issue.

CORRECTIVE ACTION

The platform release was rolled back to the previous version to restore functionality.

Keywords RCA RCA_2021.02.08 Document Upload Import raexternalrequest

Solution ID

220518101146167

Last Modified Date

05/18/2022 10:17:19 AM

Renew Date

05/18/2023

Status

Published

Author

bmagdy@uplandsoftware.com

Taxonomy

  • Mobile Commons > RCA-Root Cause Analysis

Collections

  • UMM Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2