Skip to main content

Mobile Commons: RCA - 2022/05/10 - Heavy Load - 502 Bad gateway timeout errors

MC Admin

Title

Mobile Commons: RCA - 2022/05/10 - Heavy Load - 502 Bad gateway timeout errors

Information

TITLE: Mobile Commons: Heavy Load - 502 Bad gateway timeout errors

DATE, TIME AND DURATION

START TIME: 2022/05/10 – 17:00 RESOLUTION TIME: 2022/05/10 – 18:45 DURATION: 0D 1H 45M

OVERVIEW

  • On May 10th, 2022, around 17:00 EDT, Upland Support started receiving reports from clients that there were 502 Bad Gateway errors when trying to login to the Mobile Commons platform.
  • Investigation showed that this is related to a high load on the database (DB) and the receiving queue was consistently full. In this case, it was sending notices to stop the data replication until the queries are processed.
  • This caused the DB to reject the write queries, due to these errors, which was affecting the client activity and causing the 502 errors.

ROOT CAUSE & REMEDIATION

  • ROOT CAUSE: Database queries causing table locks prevented normal platform performance and access, in addition to API requests and broadcast generation activities. This caused fast queries to wait until a timeout threshold elapsed and failed or caused queries to continue with results after a long delay. Some failed processes were retried, which added to the impact of the table lock, and would either fail again or slowly work through their task before moving on to the next queued task.
  • REMEDIATION PLAN:
    • Short term: If this situation begins to occur again, monitoring systems will be alerted and the CloudOps team will clean up the unnecessary queries on the affected database node to restore performance.
    • Long term: As part of our continued effort to provide a reliable and stable platform, we are addressing this issue by implementing safeguards against extraneous database

demand. With the development effort currently in progress and in response to this incident, the system will exist more gracefully in these scenarios, alerting a response team and immediately relieving the demand put on the database.

  • RCA ISSUE ID: [NA]

Keywords RCA RCA.5.10.2022 import document raexternalrequest

Solution ID

220523141437243

Last Modified Date

05/23/2022 02:23:35 PM

Renew Date

05/23/2023

Status

Published

Author

bmagdy@uplandsoftware.com

Taxonomy

  • Mobile Commons > RCA-Root Cause Analysis > 2022

Collections

  • UMM Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2