Title

Mobile Commons: DB Heavy Load - 502 Time-out errors - 8/16/2022

Information

TITLE: Mobile Commons – DB Heavy Load - 502 Time-out errors - 8/16/2022

DATE, TIME AND DURATION

START TIME: 2022/08/16 – 12:38 ET RESOLUTION TIME: 2022/08/16 – 14:06 ET DURATION: 0D 1H 28M

OVERVIEW

The Upland Mobile Commons team started seeing 502 time-out errors when trying to access the platform through UI on 8/16/2022 around 12:38PM ET. Our cloud-ops team started working immediately on investigating the root cause and restoring the system performance. The investigation showed that this was related to a spike in the DB load that affected the overall performance.

ROOT CAUSE & REMEDIATION

ROOT CAUSE: A large delete transaction of over 5.5M rows causing the primary database resources to be consumed over about a 2-hour window. As the large delete transaction was running, other queries were taking longer and longer to finish until they could not complete and instead just stacked up waiting.
REMEDIATION PLAN:
- Short term: After pinpointing the cause, the cloud-ops team stopped the original DB transaction and started removing the targeted data in smaller batches from the database instead of deleting at a large scale.
- Long term: The cloud team is working on adding DB alerting so we can get notified when a DB column is close to reaching the max size so we can do the necessary adjustments in advance and avoid any failures.
RCA ISSUE ID: [MC-10782]

Keywords RCA MC_RCA 2022.8.16

Solution ID

220902093209233

Last Modified Date

10/17/2022 10:00:36 AM

Renew Date

09/02/2023

Status

Published

Author

bmagdy@uplandsoftware.com

Taxonomy

Collections

Upland RightAnswers Portal - Version 2024R2