Title
Mobile Commons: RCA - 2021/04/12 - Database outage
Information
Title: Mobile Commons Outage – 4/12/2021
Incident Number: MC-20210412 Ticket/Request Number: #MC-8389 Date: 15 Apr
2021
DESCRIPTION
The purpose of this Root Cause Analysis (RCA) is to provide the factors that caused a Mobile Commons platform outage for several hours on April 12, 2021.
ISSUE DURATION
The issue began at 3:28 AM EST on Monday, April 12th and was resolved for customers at 3:46 PM EST.
DETAIL
The Mobile Commons Engineering staff received automated alerts of database errors beginning at 3:28AM on Monday, April 12. The errors were specific to a large spike of requests to the database which caused traffic unable to write new records to the DB making the entire database cluster locked and unusable. To resolve the issue the team rebuilt the individual database cluster nodes to prevent data loss. Once the nodes were restored to the cluster the services were brought back up and monitored for overall performance.
ROOT CAUSE
The Root Cause of the outage was found to be that the database was locked by a significant and unusual spike of data queries. This affected some of the database nodes but the redundant nodes were able to be leveraged to restore the database.
CORRECTIVE ACTION
The Mobile Commons database infrastructure was improved to use a more robust footprint that can tolerate a more extreme level of data spike.
Keywords RCA RCA_2021.04.15 document upload import raexternalrequest
Solution ID
220518095059060
Last Modified Date
05/18/2022 10:08:51 AM
Renew Date
05/18/2023
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis > 2022
Collections
- UMM Support
Upland RightAnswers Portal - Version 2024R2