Title: Mobile Commons DB Overload – 9/30/2021

Date, Time and Duration

START TIME: 2021-09-30 09:00 AM Eastern Time

RESOLUTION TIME: 2021-10-05 05:30 PM Eastern Time

Overview

On September 30, 2021, we received multiple reports around API 502 consistent time-outs errors, UI time-outs, delays in processing incoming messages, and general slowness.

Root Cause

On September 26th, our vendor, AWS, experienced an outage that impacted one of our database nodes. We immediately started rebuilding a new healthy DB node, however, the rebuild and recovery is a long-running process and took through September 30th in order to complete. On the 30th of September, we started experiencing an extremely heavy load due to the end of the month/quarter messaging traffic in addition to high traffic of complex long-running API requests. All of this overloaded the DB, which was still rebuilding the unhealthy DB node, and caused overall degraded system performance. At around 9:30 PM Eastern on September 30, the rebuild process of the new DB node was completed and the database resumed working with full power. Most of the platform features started to perform normally, however, we continued to experience extremely high traffic of very complex API requests that continued to cause intermittent 502 time-out errors with the API service through early October. Upland was able to work with customers to optimize these requests and fully restore connectivity to all customers by October 5.

Corrective Action

1-      Immediately started rebuilding a new DB node.

2-      Worked on reducing the number of services running on the DB to give priority to the essential traffic/queries, which helped a bit in reducing the load.

3-      Worked closely with our clients on their API integrations on the best ways to pull the needed data and to eliminate unnecessary complex requests that would overload the DB. This helped fully restoring the API service and stop the 502 time-out errors.