Title: Mobile Commons DB high load – Groups Slowness

Date, Time and Duration

START TIME: 2022-01-21 10:15 AM Eastern Time

RESOLUTION TIME: 2022-01-21 10:45PM Eastern Time

Overview

On January 21, 2022, we started receiving multiple reports of general platform slowness that affected broadcasts sending rates, groups creation, and UI slowness.

Root Cause

The database was under heavy load due to some transactions stuck in rollback that affected one of the DB nodes. Trying to stop these during the day would have ended up bringing down the entire DB.

Corrective Action

1-      Worked on cleaning up any unnecessary complex DB queries to free up resources and help ease the load on the DB

2-      Long-Term: Performed a DB maintenance on the same day late at night (10:45 PM EST) to replace the affected DB node and clean things up.    

3-      Additionally, we currently working on adding backend code in place to not allow triggering multiple static versions of the same dynamic group when there is a backend job already in progress for the same process. This would help prevent unnecessary complex DB queries and avoid unexpected DB heavy load.