Rant & Rave: RCA - 2022/5/30 - Platform Login Error’s
- 4hrs
Title
Rant & Rave: RCA - 2022/5/30 - Platform Login Error’s - 4hrs
Information
TITLE: R&R Platform Login Error’s (May 30th, 2022)
DATE, TIME AND DURATION
START TIME: May 30th, 2022, 6:43am BST RESOLUTION TIME: May 30th, 2022, 10:45am BST DURATION: 4 Hrs
OVERVIEW
Starting at around 6:45am (BST) on Monday the 30th of May 2022, multi project users of the R&R platform began to experience slightly degraded performance when logging into the Dashboard UI. This only effected the R&R Dashboard (Fast Feedback) tool.
Multi project users on the platform during this time would have experienced display issues where a solid box was covering the project drop down menu. Users retained the ability to login to the platform however this was not user friendly.
At around 8:45am BST on Monday the 30th of May 2022, our engineers started to roll back the deployment of the new login screen, during this process some users accessing the R&R Dashboard experienced errors and were unable to access the tool.
The rollback was completed by 10:45am BST on Monday the 30th of May 2022, and users were able to use the platform as normal.
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
On the 30th of May we deployed a change which was intended to update the login screen UI for the R&R Dashboard tool, this was an intended deployment which was subject to the standard QA process, which was conducted and deemed successful.
Upon completing the deployment, it was brought to our attention that a specific user type were experiencing unintentional UI changes. These changes were not present for internal users with SuperDB accounts which is why this was not picked up during the QA process.
Once the deployment was completed and we were able to replicate this display it was clear we needed to roll back the changes. When rolling back a deployment it is expected that users will have some degree of degraded performance when logging into the tool, unfortunately this was a rather large deployment where instead of taking a few minutes it took 45 minutes to complete the rollback.
REMEDIATION PLAN:
Additional QA measures have been implemented to ensure we are confirming any deployments are functioning correctly for both internal SuperDB user credentials as well as external non- SuperDB user configurations.
RCA ISSUE ID:
Keywords RCA
Solution ID
220608100406127
Last Modified Date
04/28/2023 01:36:08 PM
Renew Date
05/06/2025
Status
Published
Author
Taxonomy
- Rant & Rave > RCA-Root Cause Analysis > 2022
Collections
- Rant & Rave Support
Upland RightAnswers Portal - Version 2024R2