Information
Mobile Commons: RCA - 2023/9/25 - Database outage
TITLE: Mobile Commons - Database outage 2023/09/25
DATE, TIME AND DURATION
START TIME: 2023/09/25 – 09:00 ET RESOLUTION TIME: 2023/09/25 – 19:00 ET DURATION: 0D 10H 0M
OVERVIEW
- On Monday, 9/25, the Mobile Commons Operation team was alerted that the Database streamer service which is responsible for syncing the data between the main production database and the analytical database is inoperative. The team immediately started working on fixing the issue. The DB streamer was restored and the data syncing resumed at 7 PM ET.
After the data was fully synced, it was noticed that there were some data mismatches, specifically around groups. Due to this, the team continued to work on that side and got it fully synced and normally operating by 9/26 at 3 PM ET.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: The database Streamer service was inoperative.
- REMEDIATION PLAN: The engineering team is working on implementing an additional alerting and automatic restoration process so that if this ever happened again, it would recover as quickly as possible.
- RCA ISSUE ID: [MC_13044]
Keywords RCA outage database mobile-commons
Solution ID
231011105508473
Last Modified Date
10/25/2024 12:10:29 PM
Renew Date
08/01/2025
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis > 2023
Collections
- UMM Self-Service
- UMM Support
Upland RightAnswers Portal - Version 2024R2