Title: Mobile Commons DB high load – Groups Slowness

Date, Time and Duration

START TIME: 2022-01-27 10:55 AM Eastern Time

RESOLUTION TIME: 2022-01-27 3:20PM Eastern Time

Overview

On January 27, 2022, Mobile Commons Support received reports around extremely slow uploaded, static and A/B test groups.

Root Cause

The investigation showed that this slowness was related to a spike in complex insertion DB queries. Many users across the platform triggered multiple duplicate attempts to create static copies of the same dynamic groups. This added an extremely high/unnecessary load on the DB and affected the groups' creation performance across the platform.

Corrective Action

1-      Short-term: Immediately started working on cleaning up the duplicate/unwanted DB jobs and restored the normal system performance

2-      Long-Term: Added a backend code in place to not allow triggering multiple static versions of the same dynamic group when there is a backend job already in progress for the same process.