Skip to main content

Mobile Commons: RCA - 2022/08/14 - DB high load - 502 errors

MC Admin

Title

Mobile Commons: RCA - 2022/08/14 - DB high load - 502 errors

Information

TITLE: Mobile Commons | DB Heavy load – 502 time-out errors | 2022/08/14

DATE, TIME AND DURATION

START TIME: 2022/08/14 – 12:00 ET RESOLUTION TIME: 2022/08/14 – 18:00 ET DURATION: 0D 06H 00M

OVERVIEW

  • On 8/14 around 12 PM ET, the Mobile Commons cloud-ops team was alerted of a high data base (DB) load that affected one of the DB nodes that resulted in affecting the platform performance and caused time-outs in multiple platform areas.

ROOT CAUSE & REMEDIATION

  • Root Cause: The investigation showed that this was due to a few long running queries that were not responding and affected the main DB node.
  • REMEDIATION PLAN:
    • Short term: The Mobile Commons cloud-ops team started working immediately on replacing the affected DB node. After multiple unsuccessful attempts to replace the affected DB node, the team decided to re-build the full DB cluster and managed to restore the system around 6 PM ET.
    • Long term: The cloud team is actively working on a long-term infrastructure improvement process including switching to a new, more robust DB technology that would eliminate the performance issues we are currently seeing.
  • RCA ISSUE ID: [MC-10609]

Keywords RCA MC_RCA 2022.8.14 raexternalrequest

Solution ID

220902091001993

Last Modified Date

09/02/2022 09:31:46 AM

Renew Date

09/02/2023

Status

Published

Author

bmagdy@uplandsoftware.com

Taxonomy

  • Mobile Commons > RCA-Root Cause Analysis > 2022

Collections

  • UMM Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2