Waterfall: RCA - 2024/03/06 - Second Broadcast Message Sent to Production Recipients testing on Product Environment

Title

Waterfall: RCA - 2024/03/06 - Second Broadcast Message Sent to Production Recipients testing on Product Environment

Information

C,upland

Root Cause Analysis

For Ex ernal Release

TITLE: Second Broadcast Mess,age Sent to Production Recipients -Testing on Product Environment

DATE, TIME AND DURATION

START TIME: 06/03/2024 - 06:30 CST/ 1230 GMT RESOLUTION TIME: 06/03/2024 - 06:30 CST/ 12:30 GMT DURATION: OOD OOH 0.5M

OVERVIEW

• On 6 March 2024 at12:30 GMT, a broadcast message was accidentally s,ent from the number -t44 7366 731434 to two lists that contained 1,777 recipients. Our team has identified the messages were sent as the result of a test being run by the development team in the production environment. This is unfortunately the second occurrence of e1nroneously sent messages to IKeoghs.recipients: the first instance was due to a test conducted in staging; the new occunrence was due to a demonstration that was imprope ly dosed in the production environment

ROOT CAUSE & REMEDIATION ROOT CAUSE:

• While the dev team was running demonstrations, two test lists (118_test_list and

118_test_rep) were mistakenly !,eft in the production environment.

• Thes,e lists were mapped to itlhe test account in production, and the platform subs,equently sent a test message oontaining the terms "Simple BC."
• The remediation steps from the previous incident are yet to be completed, resulting in the steps that Ied to this incident
  • Audit of Existing Lists and Aocess
  • List Mana,gement and Governance

C!)upland

Root Cause Analysis

or Ex ernal Release

REMEDIATION!PLAN:

Complete all remediation1irtems from the previous inc.iden1t:

• • Audit of Existing lists andPersonnel
  • Governance .Around lists
    • List management training, best practices
    • Creating custom lists vs. pre-existing lists
    • Update prooess,es around list creation, deletion, and auditing
    • Set up permission-based rul,es to restriict adding custom numbers to staging.
    • All test messages used in testing or demonstrations will utili2:e approved verbiage in the message: "Upland Test Message"

Remecfiatfon Managemen1t

• • Create internal polides to ensure timely comp!,ete of critioal remediation tasks
  • Streamline RCA. and Remediation process to include sign-off dates and ticket IDs
  • Led by Waterfall Proj,ect Managers
• RCA.ISSUE.ID: 1483267

Keywords 2024 broadcast production testing environment RCA message

Solution ID

240313105608747

Last Modified Date

03/22/2024 09:47:47 AM

Renew Date

03/13/2025

Status

Published

Author

dzala@uplandsoftware.com

Taxonomy

• Waterfall > RCA-Root Cause Analysis > 2023

Collections

• Waterfall Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2