Skip to main content

Waterfall: RCA - 2024/08/28 - 9Rooftops Automated Broadcast Issue

MC Admin

Information

TITLE: 9Rooftops Automated Broadcast Issue

DATE, TIME, AND DURATION

START TIME: 2024/08/28 – 09:00 AM EST. RESOLUTION TIME: 2024/08/28 – 4:50 PM EST.DURATION: 0D 7H 50M

OVERVIEW

  • The automated scheduled broadcasts for the customer, 9Rooftops, did not execute asexpected.

ROOT CAUSE & REMEDIATION

ROOT CAUSE:

The issue was a database connection error in the web service related to the client, 9Rooftops. The web service was configured to connect to a pre-MySQL-upgrade DNS host (old host). This configuration continued to work until the old host was active, up toAugust 27, 2024.

Post-August 27, 2024, the old host was decommissioned and deleted as part of the migration process to a new database host that was completed over a month ago.

The old host was kept temporarily as a backup to manage any unforeseen failures. Once the old host was deleted, the web service could no longer establish a databaseconnection, leading to the failure of the scheduled broadcasts.

REMEDIATION PLAN:

To resolve the issue, the following actions were taken:

    • The Upland Services team has manually launched the failed broadcasts.
    • The Upland Engineering team has updated the database host configurations in the web service to point to the correct new DNS host, restoring the automated

process'sfunctionality.

PREVENTION OF FUTURE RECURRENCE:

    • Update Management: Ensure that any updates to database configurations arereflected across all related web service files promptly.
    • Enhanced Alerting and Logging: Improve alerting and logging mechanisms within the client services application. This will include fixing outdated and non-functionalemail alert systems to ensure timely notifications of any service issues.
    • Documentation: Maintain comprehensive documentation of all active web serviceswithin the application to ensure clarity and prevent misconfigurations.
  • RCA ISSUE ID: [WF-11086]

Keywords Waterfall: RCA - 2024/08/28 - 9Rooftops Automated Broadcast Issue

Solution ID

240910053547573

Last Modified Date

09/10/2024 05:40:05 AM

Renew Date

09/10/2025

Status

Published

Author

amuhaz@uplandsoftware.com

Taxonomy

  • Waterfall > RCA-Root Cause Analysis > 2024

Collections

  • Waterfall Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2

Related articles