Title
Rant & Rave - RCA: 2024/22/4 - Users unable to log in to Rant & Rave
Information
DATE, TIME AND DURATION
START TIME: 08:00 – 22nd April, 2024
RESOLUTION TIME: 09:58 – 22nd April, 2024
DURATION: 1hr, 58mins
OVERVIEW
During the above period customers were unable to log in to the Rant & Rave platform. This issue only impacted logging in. All background processes continued as normal.
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
Prior to this issue we had an incident which required us to rebuild the replica databases. The replica and master databases had become out of sync. The customer facing URL was temporarily pointed at master while new replicas were rebuilt.
These replica databases that were spun up didn’t have the necessary security groups assigned to them. This meant that our permissions API couldn’t connect to them. Customers would try to log in and the permissions API would try to connect to the replica database to verify credentials. At that point it couldn’t connect and as such customers couldn’t log in.
REMEDIATION PLAN:
In the short term the security group for these databases was simply amended to allow for the correct services. This resolved the issue as far as users being able to log in again.
In the medium term if this was to happen again the Cloud team will have to ensure that they amend the default security group settings so that this does not reoccur.
The long term remediation is that the Cloud team have raised a ticket with Security asking for the default permissions to be updated. This will mean in future no additional actions will be required.
RCA ISSUE ID:
1513527
Keywords rant rave rca
Solution ID
240523113919943
Last Modified Date
05/23/2024 11:39:20 AM
Renew Date
05/23/2025
Status
Published
Author
Taxonomy
- Rant & Rave > RCA-Root Cause Analysis
Collections
- Rant & Rave Support
Upland RightAnswers Portal - Version 2024R2