Skip to main content

Mobile Commons: RCA - 9/30/2021 - MC Database overload

MC Admin

Title

Mobile Commons: RCA - 9/30/2021 - MC Database overload

Information

Title: Mobile Commons DB Overload – 9/30/2021

DATE, TIME AND DURATION

START TIME: 2021-09-30 09:00 AM Eastern Time RESOLUTION TIME: 2021-10-05 05:30 PM Eastern Time

OVERVIEW

On September 30, 2021, we received multiple reports around API 502 consistent time-outs errors, UI time-outs, delays in processing incoming messages, and general slowness.

ROOT CAUSE

On September 26th, our vendor, AWS, experienced an outage that impacted one of our database nodes. We immediately started rebuilding a new healthy DB node, however, the rebuild and recovery is a long-running process and took through September 30th in order to complete. On the 30th of September, we started experiencing an extremely heavy load due to the end of the month/quarter messaging traffic in addition to high traffic of complex long- running API requests. All of this overloaded the DB, which was still rebuilding the unhealthy DB node, and caused overall degraded system performance. At around 9:30 PM Eastern on September 30, the rebuild process of the new DB node was completed and the database resumed working with full power. Most of the platform features started to perform normally, however, we continued to experience extremely high traffic of very complex API requests that continued to cause intermittent 502 time-out errors with the API service through early October. Upland was able to work with customers to optimize these requests and fully restore connectivity to all customers by October 5.

CORRECTIVE ACTION

  1. Immediately started rebuilding a new DB node.
  2. Worked on reducing the number of services running on the DB to give priority to the essential traffic/queries, which helped a bit in reducing the load.
  3. Worked closely with our clients on their API integrations on the best ways to pull the needed data and to eliminate unnecessary complex requests that would overload the DB. This helped fully restoring the API service and stop the 502 time-out errors.

Keywords RCA Upload document import process RCA_90.30.2021 raexternalrequest

Solution ID

220518083028053

Last Modified Date

05/18/2022 09:16:03 AM

Renew Date

05/18/2023

Status

Published

Author

bmagdy@uplandsoftware.com

Taxonomy

  • Mobile Commons > RCA-Root Cause Analysis > 2022

Collections

  • UMM Support

Go to Original Solution

Upland RightAnswers Portal - Version 2024R2