Title
Rant & Rave - RCA: 2024/4/19 - Failure of database sync causing caused delay in feedback posting to dashboard.
Information
DATE, TIME AND DURATION
START TIME: 16:45 - 19th April, 2024
RESOLUTION TIME: 03:00 - 22nd April, 2024
DURATION: 58hrs 15mins
OVERVIEW
During the above period we experienced an issue which caused new feedback to not be visible in the Rant & Rave dashboard. Feedback created up the incident remained visible.
This was identified as a database sync issue between our Master and Read databases. Once the issue was resolved all feedback was visible again within the Dashboard. There was no loss of data due to this issue.
ROOT CAUSE & REMEDIATION
ROOT CAUSE:
Every few seconds our Master database generates a file which our Replica/Read databases pick up. This incident was caused due to our replica database failing to pick up that file. The replica won’t move onto the next file until it has picked up the previous one and as such the sync was broken.
When users log into the platform they see the read database. Because of this when they logged in, they were seeing feedback up until the file failed to be picked. Any new feedback that came in was not visible.
The exact reason why the replica database failed to pick up this file is unknown at this time.
REMEDIATION PLAN:
In the short we directed the platform URL to the master database rather than the read one. This meant that customers began seeing up to date information again and, from their POV, the issue was resolved.
After this our cloud team began to spin up new replicate databases to replace the ones which had failed to sync.
These take some time to spin up and as such the replica databases were online and in rotation at 3am, 6am and 10am respectively on the 22nd of April, 2024. On the evening of the 22nd we took the took the master database out of
the read group so that customers we again only being directed to read.
For long term mitigation a ticket has been raised to add monitoring of the master and read sync so that we are alerted to a failure of this in future. The Cloud team are in ongoing discussions with Xtivia about the failure of the file pickup. There has been some early conversation around extending the files from two hours to two days to avoid this in future but that is ongoing.
RCA ISSUE ID:
1513194
Keywords Rant & Rave RCA
Solution ID
240523110000510
Last Modified Date
05/23/2024 11:01:20 AM
Renew Date
05/23/2025
Status
Published
Author
Taxonomy
- Rant & Rave > RCA-Root Cause Analysis
Collections
- Rant & Rave Support
Upland RightAnswers Portal - Version 2024R2