Title
Rant & Rave: RCA - 2022/4/5 - Platform outage and instability - 32.5hrs
Information
TITLE: R&R Platform Performance Issues (April 5th – 6th, 2022)
DATE, TIME AND DURATION
START TIME: April 5th, 2022, 8am BST RESOLUTION TIME: April 6th, 2022 – 4.30pm BST DURATION: 32.5 Hrs
OVERVIEW
Starting at around 8am (BST) on Tuesday the 5th of April 2022, users of the R&R platform began to experience significantly degraded performance. This affected all R&R tools such as Fast Feedback, Discover, ProComms Desktop and User Management.
Users on the platform during this time would have experienced timeout errors, extreme slowness when navigating between areas of the platform and widgets/tables not loading fully. These issues were intermittent and particularly noticeable during peak UK business hours.
Scheduled email solicitations went out as normal. SMS workflows experienced some slight delays but were still ultimately successful. Our databases did not suffer any data loss.
Many API calls failed during this period, especially during peak UK business hours. Because our servers were unable to respond quickly enough the API calls timed out and would not have been received by us. The impact to clients would be that triggered surveys were never received by us or sent on to recipients. Customers wishing to ensure these all succeeded will have had to manually refire these failed API calls once the system was stable.
At around 4.30pm on Wednesday the 6th of April 2022, our engineers were able to deploy several infrastructure changes to relieve the problem and performance improved significantly. There were no further reports of timeouts or slowness, and users were able to use the platform as normal
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
On the 1st of April we deployed a change which was intended to refresh the dashboard for a specific customer every two minutes. This was an intended deployment of requested functionality but that was intended to only impact a single customer. Unfortunately, human error resulted in this change being deployed for all customers.
The impact of this change was not felt fully until Tuesday the 5th of April. The number of dashboard refresh requests we receive is typically very consistent and we maintain a 50%-100% additional capacity for this area of the platform. The number of dashboard refresh requests began to quickly outstrip this capacity however and the impact to clients began.
As our systems began to struggle with capacity and slowness our own Health Checks also began to exacerbate the problem. These checks would typically mark slow running machines as ‘sick’ and take them out of rotation accordingly so that they can be investigated. The unprecedented increase in demand was resulting in numerous machines being marked as sick and being taken out of rotation when they could have continued to function, albeit slowly.
Simply rolling back the change that we made was among the first things we did to resolve this issue. While this reverted the code in production it did not revert the database changes to dashboard refresh which exist on a user level. This hampered our investigation and caused further delays as we believed the deployment was unrelated to the ongoing issues.
REMEDIATION PLAN:
The refresh feature has now been removed entirely from the platform. We hope to release this as an optional feature to customers in the future in a controlled manner.
Our health checks have been optimized so that we have more control over the flagging and removing of sick machines. This ensures that if a similar issue was to occur again that we would be able to control more easily whether we decide to keep slow but functional machines in rotation.
Additional monitoring and reporting tools have been put in place so that we can pre-emptively resolve potential issues like this in the future before they have an impact on customers. We can now more accurately pinpoint the direct cause of spikes in traffic. Were a similar incident to happen again in the future we would know what process is causing the issue and be able to deal with it immediately.
We do not believe that an increase in capacity is needed but we are implementing processes to enable us to more quickly increase capacity in the future if the need did arise.
While there is no mechanism in place that would allow us to capture API calls that have timed out we believe the changes made above should ensure that this is not needed in future. The health check optimization and increased monitoring should ensure that API calls would never fail due to timeouts.
RCA ISSUE ID:
Keywords RCA
Solution ID
220601031726587
Last Modified Date
06/14/2022 09:45:12 AM
Renew Date
06/14/2023
Status
Published
Author
Taxonomy
- Rant & Rave > RCA-Root Cause Analysis > 2022
Collections
- Rant & Rave Support
Upland RightAnswers Portal - Version 2024R2