Title
Mobile Commons: RCA - 2022/01/21 - Database Load – Groups Slowness
Information
Title: Mobile Commons DB high load – Groups Slowness
DATE, TIME AND DURATION
START TIME: 2022-01-21 10:15 AM Eastern Time RESOLUTION TIME: 2022-01-21 10:45PM Eastern Time
OVERVIEW
On January 21, 2022, we started receiving multiple reports of general platform slowness that affected broadcasts sending rates, groups creation, and UI slowness.
ROOT CAUSE
The database was under heavy load due to some transactions stuck in rollback that affected one of the DB nodes. Trying to stop these during the day would have ended up bringing down the entire DB.
CORRECTIVE ACTION
- Worked on cleaning up any unnecessary complex DB queries to free up resources and help ease the load on the DB
- Long-Term: Performed a DB maintenance on the same day late at night (10:45 PM EST) to replace the affected DB node and clean things up.
- Additionally, we currently working on adding backend code in place to not allow triggering multiple static versions of the same dynamic group when there is a backend job already in progress for the same process. This would help prevent unnecessary complex DB queries and avoid unexpected DB heavy load.
Keywords RCA_ RCA_2022.01.21 import upload document raexternalrequest
Solution ID
220518093055657
Last Modified Date
05/18/2022 09:35:51 AM
Renew Date
05/18/2023
Status
Published
Author
Taxonomy
- Mobile Commons > RCA-Root Cause Analysis > 2022
Collections
- UMM Support
Upland RightAnswers Portal - Version 2024R2