Title
Rant & Rave: RCA - 2022-06-10 - Outage 2hrs
Information
TITLE: Rant & Rave Platform Outage – June 10th, 2022
DATE, TIME AND DURATION
START TIME: 10/6/2022 – 5pm BST RESOLUTION TIME: 10/6/2022 = 7pm BST DURATION: 2hrs
OVERVIEW
On the 10th of June, 2022, the Rant & Rave platform experienced an outage that lasted around two hours. During this outage access to all Rant & Rave systems would have been affected. Users trying to access our Fast Feedback, Discover, ProComms or User Management services would have received an error message. No data was lost during this event, but any API calls made during the outage period would have failed.
On resolution of the incident all access was restored and there was not further impacted to customers with the exception of API calls that had failed which may have needed to be re-run.
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
One of our firewall devices failed and became unresponsive, which caused communication between our external load balancer and internal applications to fail in turn. We are unable to determine the exact failure with the firewall device.
REMEDIATION PLAN:
As we were unable to resolve the issue with our firewall device our operations team have instead rerouted traffic around the affected device and replaced its services with alternative Amazon services to ensure that security is unaffected.
We have decided as the long-term solution to this issue to keep these services running via Amazon as this provides as secure a service while also providing us with greater robustness.
RCA ISSUE ID:
1044987
Keywords rca outage
Solution ID
220707042744730
Last Modified Date
07/07/2022 04:28:12 AM
Renew Date
07/07/2023
Status
Published
Author
Taxonomy
- Rant & Rave > RCA-Root Cause Analysis > 2022
Collections
- Rant & Rave Support
Upland RightAnswers Portal - Version 2024R2