Information

TITLE: Waterfall Carrier Lookup Failure

DATE, TIME AND DURATION

START TIME: 2024/08/10 – 07:30 UTC

RESOLUTION TIME: 2024/08/13 – 18:30 UTC

OVERVIEW

• The issue was caused by a carrier lookup failure in the waterfall platform due to incorrect OpenMarket (OM) carrier lookup credentials. These credentials became invalid after OM updated their lookup API credentials, leading to disruptions in the UK waterfall platform. The problem began on August 10, 2024, at 07:30 UTC, and was resolved on August 13, 2024, at 18:30 UTC.

ROOT CAUSE & REMEDIATION

ROOT CAUSE:

• • Incorrect OpenMarket carrier lookup credentials in the waterfall platform.
  • The credentials in the waterfall platform got invalid after OM updated the credentials for their lookup APIs.

REMEDIATION PLAN:

• Short-term plans:
  • Credentials modification
  • Replay of the failed solicitations as per our clients' request
• Long-term plans:
  • Possibility of having a secondary set of credentials that can be used when the primary fails
  • Put the logic to alert the team when a credential failure happens
  • Work on using a hashing mechanism/vault to store the credentials and not embed them in code plainly
  • Work on separating the credentials/configs from the code so that any config change does not require a code deployment.
• RCA ISSUE ID: MAPI-11000

Keywords Waterfall: RCA - 2024/08/10 - Carrier LookUp Failure

Solution ID

240823123345557

Last Modified Date

08/23/2024 12:33:48 PM

Renew Date

08/23/2025

Status

Published

Author

amuhaz@uplandsoftware.com

Taxonomy

• Waterfall > RCA-Root Cause Analysis > 2024

Collections

• Waterfall Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2