Title:  Mobile Commons Outage – 4/12/2021

The purpose of this Root Cause Analysis (RCA) is to provide the factors that caused a Mobile Commons platform outage for several hours on April 12, 2021.

The issue began at 3:28 AM EST on Monday, April 12th and was resolved for customers at 3:46 PM EST.

The Mobile Commons Engineering staff received automated alerts of database errors beginning at 3:28AM on Monday, April 12. The errors were specific to a large spike of requests to the database which caused traffic unable to write new records to the DB making the entire database cluster locked and unusable. To resolve the issue the team rebuilt the individual database cluster nodes to prevent data loss. Once the nodes were restored to the cluster the services were brought back up and monitored for overall performance.

The Root Cause of the outage was found to be that the database was locked by a significant and unusual spike of data queries. This affected some of the database nodes but the redundant nodes were able to be leveraged to restore the database. 

The Mobile Commons database infrastructure was improved to use a more robust footprint that can tolerate a more extreme level of data spike.