Title: Mobile Commons DB high load – intermittent UI time outs

Date, Time and Duration

START TIME: 2022-01-14 3:20pm PM Eastern Time

RESOLUTION TIME: 2022-01-14 4:20 PM Eastern Time

Overview

On January 14, 2022, Mobile Commons Support received reports of intermittent UI time-out errors, delays in mConnect calls, and general platform slowness.

Root Cause

On Jan 14, 2022, we started receiving an unexpected extremely high volume of API calls from our 3rd party vendor related to triggering the mConnect/outbound phone calls from the platform. These requests started to pile up and cause locking on the DB. During that time, the system was also trying to process high volume of MMS DLRs which added more load on the DB.

Corrective Action

The team started working immediately on cleaning any complex unnecessary queries on the DB. Additionally, we worked on restarting the passenger processes on the app servers once they have attempted to process a request for more than 5min to keep things running.