Skip to main content

Waterfall: RCA -- Waterfall Broadcast Outage

MC Admin

Title

Waterfall: RCA -- Waterfall Broadcast Outage

Information

TITLE: Waterfall Broadcast Outage

DATE, TIME AND DURATION

START TIME: March 1, 2024 – 00:00 Central Time

RESOLUTION TIME: March 2, 2024 – 01:10 Central Time

DURATION: 1D 1H 10M

OVERVIEW

  • On March 1st, 2024 at 5:17am CT, Waterfall customers began reporting that scheduled SMS broadcasts for the morning had failed to send. Rescheduling or attempting to resend led to further errors. Our engineers identified the issue and deployed a hotfix at 10am CT to resolve the rescheduling issues, allowing the system to process rescheduled broadcasts and send them successfully.

The above fix successfully addressed the issues with scheduled broadcasts; however, API- driven broadcasts continued to fail. Our technical teams continued troubleshooting and eventually deployed a larger fix on March 2 at 1:00am CT that specifically addressed the API-driven broadcast issues.

ROOT CAUSE & REMEDIATION

ROOT CAUSE:

    • The broadcast disruptions occurred because of an update that was released on March 1st at 12am CT.
    • The release added throttling parameters to enhance broadcast capabilities. Unfortunately, the release did not successfully update two key components: existing scheduled broadcasts and API-based broadcasts.
    • During QA, only new broadcasts were tested but API and scheduled broadcasts were not tested.
    • Internal reporting and testing failed to show the necessary components during planning, which led to the update being released without proper testing.

REMEDIATION:

    • Audit of all broadcast activity, use cases, and types to ensure visibility around all known environments. Each case will be reviewed carefully during release planning and execution.
    • Default all internal reporting to show pertinent platform data to ensure all current / future kinds of activity are captured during planning.
    • Audit of overall Waterfall traffic to improve governance and resilience around release schedules and deployments. o Modified QA & testing process will include all use cases and environments established by the audits discussed above. o Scheduled broadcasts will be tested thoroughly by creating a scheduled broadcast internally to confirm prior to releases.
    • Create backward compatibility of APIs when modifying existing API endpoints.
    • Adopt best practices around creating new API endpoints rather than modifying existing endpoints.
  • RCA ISSUE ID: 1480612

Keywords broadcast rca outage

Solution ID

240312144919263

Last Modified Date

03/12/2024 03:11:00 PM

Renew Date

03/12/2025

Status

Published

Author

tomar@uplandsoftware.com

Taxonomy

  • Waterfall > RCA-Root Cause Analysis

Collections

  • Waterfall Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2

Related articles