Mobile Commons: RCA - 2022/04/19 - Database Load – 502 UI time-out errors – Slowness with uploaded groups
Title
Mobile Commons: RCA - 2022/04/19 - Database Load – 502 UI time-out errors – Slowness with uploaded groups
Information
TITLE: [Database Load – 502 UI time-out errors – Slowness with uploaded groups ]
DATE, TIME AND DURATION
START TIME: [2022/04/19 – 10:41] RESOLUTION TIME: [2022/04/19 – 13:13] DURATION: 0D 2H 32M
OVERVIEW
- On April 19, 2022, around 11:20 AM EST, we started receiving reports around slow uploaded groups and 502 HTTP time out errors when trying to login to the platform.
ROOT CAUSE & REMEDIATION
- ROOT CAUSE: 10:41-10:42 EST one of the DB nodes was disconnected from the network and momentarily interrupted all cluster nodes. The nodes did resync within a few seconds, and it took 3-4 minutes for the main node which was affected to be ready to accept any new DB queries. At 11:25EST, the delivery reports for recent MMS messages spiked and added more load to the DB, which started causing 502 UI time out error timeouts in the UI.
- REMEDIATION PLAN: The CloudOps team updated the Web load balancers to separate web traffic from other sources to eliminate the 502-time out errors. The team continued to work on cleaning up all the bad and stuck DB requests to reduce the unnecessary DB load and restore the system to full performance.
- RCA ISSUE ID: [MC-9858]
Keywords RCA RCA_4.19.2022 upload document import process
Solution ID
220505155439253
Last Modified Date
05/06/2022 01:11:16 PM
Renew Date
05/05/2023
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis > 2022
Collections
- UMM Support
Upland RightAnswers Portal - Version 2024R2