Title
Waterfall: RCA - 2024/04/04 - Internal Server Errors after Mobile API Release
Information
Gupland
Root Cause Analysis
or External Release
TITLE: Internal Serv,er Errors after Mobile API R,elease
DATE, TIME AND DURATION
STARTTIME: 2024-04-04 02:00 UTC RESOLUTION TIME: 2024-:04-0403:15 UTC DURATION: OllHI15M
OVERVIEW
The Upland development team discovered that during a production version 1re ease to the Mobil,e API platform, a plarrned databas,e migration led to an unexp,ected delay. This delay caused all incoming Message API req,uests to fail with an HTTP 500 response and were not sent out.
The most recenit planned Mobile API releas,e included a necessary database migration to suprport new upcoming functionality. This migration was reviewed p1rior and was exrpected to complete in a few minutes time, however, this database migration rprncess became stuck because of an existing large task running on the same table at the same time. Each on its own is not an issue, but due to the two prooesses running simultaneously, it caus,ed the prooessing issue which took time to understand and resolve properly.
ROOT CAUSE & REMEDIIATION
- ROOTCAUSE:
o There was a large task being run against the database while a deployment was released for the Mobille APIIand caused the database migration to delay. Onoe the previously large task was recogniz,ed to be the cause of the database migration, the CloudOps team stopped the taslk and allowed the database migration to complete within seconds and complete the p,lanned deplloyment.
Keywords RCA internal server mobile api release
Solution ID
240419052800860
Last Modified Date
04/19/2024 11:52:19 AM
Renew Date
04/19/2025
Status
Published
Author
Taxonomy
- Waterfall > RCA-Root Cause Analysis > 2023
Collections
- Waterfall Support
Upland RightAnswers Portal - Version 2024R2