TITLE: Mobile Commons | DB Heavy load – 502 time-out errors | 2022/08/14

Date, Time and Duration

START TIME: 2022/08/14 – 12:00 ET

RESOLUTION TIME: 2022/08/14 – 18:00 ET

DURATION: 0D 06H 00M

Overview

• On 8/14 around 12 PM ET, the Mobile Commons cloud-ops team was alerted of a high data base (DB) load that affected one of the DB nodes that resulted in affecting the platform performance and caused time-outs in multiple platform areas.

Root Cause & remediation

• Root Cause: The investigation showed that this was due to a few long running queries that were not responding and affected the main DB node.

• REMEDIATION PLAN:

-          Short term: The Mobile Commons cloud-ops team started working immediately on replacing the affected DB node. After multiple unsuccessful attempts to replace the affected DB node, the team decided to re-build the full DB cluster and managed to restore the system around 6 PM ET.

-          Long term: The cloud team is actively working on a long-term infrastructure improvement process including switching to a new, more robust DB technology that would eliminate the performance issues we are currently seeing.

• RCA ISSUE ID: [MC-10609]