Skip to main content

Waterfall: ATOS SMS Delivery Failure Due to Third-Party Aggregator (InfoBip) Issue

MC Admin

Waterfall: ATOS SMS Delivery Failure Due to Third- Party Aggregator (InfoBip) Issue

Information

Issue

Customer ATOS reported an SMS service outage on Saturday morning, with a configuration issue from Infobip (third-party aggregator) cited as the cause for SMS delivery failure. The customer expressed significant concern over the repeated issues, including an earlier service outage and the security implications surrounding the problem. They requested an RCA, further details on the cause, and security protocols involved.

Customer Request:

  • Detailed information on the cause of the incident and the fix.
  • Clarification on whether the issue was related to previous incidents or workaround actions.
  • Request for an RCA and details on customer impact (SMS failures).
  • Questions around security protocols, protections, and heightened monitoring in place with Upland and

Infobip.

Investigation and Resolution Steps:

Incident Identification and Response:

    • The issue was initially detected by a PagerDuty alert triggered on Saturday morning.
    • The team quickly began addressing the issue before ATOS reached out.

Root Cause Analysis (RCA) and Communication:

    • The investigation identified that the failure was due to an issue with a third-party aggregator (Infobip).
    • A fix was deployed on their side to restore SMS traffic.

Customer Impact and Failed SMS Counts:

Security Concerns and Monitoring:

    • The customer raised security concerns about the breach and asked for clarification on the protection and monitoring in place for the service.
    • The incident involved access to a third-party endpoint for sending SMS messages using Upland credentials.
    • No access to Upland’s estate was detected, and no customer data was compromised.
    • Monitoring immediately identified the issue, and Upland stopped all access to Infobip until credentials could be updated and the potential threat mitigated.

Fix Deployment and Remediation Actions:

    • A hot failover mechanism was introduced as part of the fix. This feature ensures that in the event of credential issues, the system switches to backup credentials to prevent further service disruption.
    • The solution was tested in staging and successfully verified.
    • Further investigation and updates were shared with the customer, and a deployment schedule for remediation actions was set.

Ongoing Monitoring and Updates:

    • The customer was updated on the monitoring process and confirmed that everything was functioning as expected.
    • The monitoring process showed no errors, and the new measures were confirmed to be successful.

Resolution

  • The root cause was identified and fixed, with the issue traced back to Infobip, a third-party SMS aggregator.
  • Upland’s team deployed a fix and enhanced security measures to prevent further disruptions.
  • The customer’s concerns were addressed, and the RCA was shared.
  • The ticket was closed after the customer confirmed that all questions were answered and no further action was required.

Keywords Waterfall: ATOS SMS Delivery Failure Due to Third-Party Aggregator (InfoBip) Issue

Solution ID

250131121631517

Last Modified Date

01/31/2025 12:23:05 PM

Renew Date

01/31/2026

Status

Published

Author

amuhaz@uplandsoftware.com

Taxonomy

    • Waterfall > API
    • Waterfall > RCA-Root Cause Analysis
    • Waterfall > Messaging

Collections

    • Waterfall Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2

Related articles