Information

TITLE: Atos - Mobile API SMS Outbound Issues – July 2024

DATE, TIME AND DURATION

START TIME: 8th of July, 2024 – 18:37 BST RESOLUTION TIME: 9th of July, 2024 – 10:15 BST DURATION: Approx. 15 HRS 38 MIN

OVERVIEW

Starting at 18:37 BST on Monday, July 8th, 2024, an issue occurred with the Mobile API SMS service where SMS traffic was not being passed on to the 3rd party aggregator.
The issue was fixed at 10:15 BST on Tuesday, July 9th, 2024. All new messages sent from that point on were sent successfully.

ROOT CAUSE & REMEDIATION

ROOT CAUSE:

- Upland disabled a user whose credentials were used in the connection between MAPI and our 3rd party aggregator. This action stopped messages from being successfully sent to our aggregator, which caused delivery failures to recipients.
- The user account in question was associated with an email address of an Upland employee who is no longer present, which led to the decision to disable the user.
- Internal monitoring did not initially detect this issue, as our monitors were not set up for invalid authentications, which unfortunately prevented any proactive communication regarding the issue to Atos.

REMEDIATION:

- The MAPI development team has worked with our aggregator to update the disabled account and all others used for live traffic to instead be connected to a generic email address. This address is clearly labeled as such to prevent issues in the future.
- The MAPI development team will continue investigating long-term solutions around credential management with our aggregator accounts involved in sending messages.
- The team will set up additional monitoring for the connection between MAPI and our third-party aggregator. While monitoring exists for message failures and failed requests to our aggregator, specific alerting on invalid authentication was discovered to be a gap.

MAPI did not have alerting for the corresponding HTTP failure codes around authorization/authentication, which will be added.

- Going forward, the new monitoring will provide critical alerts around authentication issues (in addition to send failures) which will allow our internal teams to proactively communicate to customers in the event of such failures.
- In addition to monitoring, the internal Upland support teams (like Rant & Rave) have set up a notification system that will escalate any SMS issues to the MAPI support team immediately. Doing so will trigger incident management activities and allow proactive communication around critical issues.
- The MAPI support team will ensure that all key personnel are subscribed to the relevant status page for real-time updates on service availability during outages, in addition to upcoming platform releases and updates: https://mobileapi.site24x7statusiq.com.
RCA ISSUE ID: MAPI-1586 / MAPI-1591

Keywords Mobile API SMS Outbound Issue Waterfall RCA MAPI

Solution ID

240805115458987

Last Modified Date

08/05/2024 11:54:58 AM

Renew Date

08/05/2025

Status

Published

Author

amuhaz@uplandsoftware.com

Taxonomy

Collections

Upland RightAnswers Portal - Version 2024R2