TITLE: Mobile Commons | DB Heavy load – 502 time-out errors | 2022/07/19

Date, Time and Duration

START TIME: 2022/07/19 – 11:20 ET

RESOLUTION TIME: 2022/07/23 – 14:34 ET

DURATION: 0D 3H 14M

Overview

• The Upland Mobile Commons team started seeing extreme platform slowness and intermittent 502 time-out errors on 7/19/2022 around 11:20 ET. Our cloud team started working immediately on investigating the root cause and restoring the system performance. The investigation showed that this was related to a heavy database load that affected one of the database nodes.

remediation

• REMEDIATION PLAN:

-          Short term: The cloud team started working immediately on replacing the affected DB node and restart all the backend processes to restore the platform performance.

-          Long term: The cloud team is actively working on a long term infrastructure improvement process including switching to a new, more robust DB technology that would eliminate the performance issues we are currently seeing.

• RCA ISSUE ID: [MC-10609]