Waterfall: RCA - 2024/02/23 - Broadcast Message Sent to Production Recipients during Testing on Stage Environment
Title
Waterfall: RCA - 2024/02/23 - Broadcast Message Sent to Production Recipients during Testing on Stage Environment
Information
TITLE: Broadcast Message Sent to Production Recipients during Testing on Sta,ge Environment
DATE, TIME AND DURATION
START TIME: [2024/02/23 - 06:07:09] RESOLUTION TIME: [2024/02/23 - 06:07:43] DURATION: OD OH OM 34S
OVERVIEW
On the 28:rd of February 2024, at around 06:07 AM UK time, during testing on the Stag1e environment, a broadcast message was accidentally sent from 44 7860 016113 to several! lists in the staging environment. Some of these lists contained production recipients and the messa;ge was sent to a totail of 4,666 phone numbers, of which 1,414 belonged to Keoghs production recipients. After receiving1this information from Keoghs, the dev team investi91a1ted and discovered thait the message was sent as a
result of □1 mainual error.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: The root cause of the incident was identified as a gap within the testing process. Specifically, the oversight occurred because the dev team missed removing production numbers from the staging environment once specific tests wascompleted, even though they were onlly meant to exist temporarily. Additionally, our investigation revealed that Keoghs numbers were added to staging back in October 2023 as a result of the validation process for Zendesk ticket 13880,22.
Keywords 2024 broadcast message stage environment RCA
Solution ID
240304053319270
Last Modified Date
03/22/2024 06:15:44 AM
Renew Date
03/04/2025
Status
Published
Author
Taxonomy
- Waterfall > RCA-Root Cause Analysis > 2023
Collections
- Waterfall Support
Upland RightAnswers Portal - Version 2024R2