Title
Rant & Rave: RCA - 2022-06-10 - Outage and File Drop Issue
Information
TITLE: Rant & Rave Platform Outage & File Drop Issues – June 10th, 2022
DATE, TIME AND DURATION
START TIME: 10/6/2022 – 5pm BST RESOLUTION TIME: 14/6/2022 = 5pm BST DURATION: 4 days
OVERVIEW
On the 10th of June, 2022, the Rant & Rave platform experienced an outage that lasted around two hours. During this outage access to all Rant & Rave systems would have been affected. Users trying to access our Fast Feedback, Discover, ProComms or User Management services would have received an error message. No data was lost during this event but any API calls made during the outage period would have failed.
On resolution of the incident all access was restored fully for the majority of customers. Sky however continued to experience issues with file drops not being successfully picked up by the Rant & Rave system. The cause of this was determined on the 14th of June and file drops were successful from that point on. Over the course of the next two days the file drops that failed were replayed successfully.
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
One of our firewall devices failed and became unresponsive, which caused communication between our external load balancer and internal applications to fail in turn. We are unable to determine the exact failure with the firewall device. This issue caused the initial two hour outage.
Remediation work that was carried out failed to take into account the custom routing that Sky had set up for SFTP file drops. This was set up in order to deal with Sky’s dropping of encrypted files. Files were being placed successfully to our server but were then never routed to the new service.
REMEDIATION PLAN:
As we were unable to resolve the issue with our firewall device our operations team have instead rerouted traffic around the affected device and replaced its services with alternative Amazon services to ensure that security is unaffected.
We have decided as the long-term solution to this issue to keep these services running via Amazon as this provides as secure a service while also providing us with greater robustness.
With regards to custom file routing, this was spotted by our Operations team and the routing successfully fixed on the 14th. From that point on file drops were successful again and we then replayed the ones that had failed previously. For longer term remediation a ticket has been raised to create end-to-end checks for this routing in future. This should ensure we are alerted to the custom set up failing at the same time as any more widescale outage.
RCA ISSUE ID:
1044987
Keywords RCA outage
Solution ID
220707042927840
Last Modified Date
07/07/2022 04:29:27 AM
Renew Date
07/07/2023
Status
Published
Author
Taxonomy
- Rant & Rave > RCA-Root Cause Analysis > 2022
Collections
- Rant & Rave Support
Upland RightAnswers Portal - Version 2024R2