Information

TITLE: Waterfall UK List Upload Issue – March 2025

START TIME: 28th March 2025, at 11:30 AM UTC RESOLUTION TIME: 28th March 2025, at 4:30 PM UTC DURATION: 5 Hours

On March 28, 2025, at 11:30 AM UTC, a job scheduler was overwhelmed by a large number of uploads, blocking new jobs and affecting uploads and messaging. To mitigate the issue, the long-running uploads were moved to an isolated server. A dedicated server will handle future uploads, scheduled during off-business hours, with team collaboration ensuring smooth operations. This issue only impacted the UK environment, and the US environment was not impacted.

ROOT CAUSE:

- The job scheduler in JobServer1, which uses Quartz for job scheduling, was running at full capacity due to the high volume of uploads being processed in parallel.
- Due to the unexpected high load of records being uploaded, the Quartz thread pool was fully utilized, preventing the scheduler from picking up new jobs. As a result, these new jobs were stuck in a "waiting" state, unable to be executed until resources became available.
- JS1 (Job Server 1) runs 2 jobs: Uploads and Messaging. Since the server was running at full capacity, both the jobs failed to process, and this impacted the message delivery along with Uploads

REMEDIATION PLAN:

- - Short Term Plan:
    - All long-running uploads were delegated to an isolated server.

- - - Going forward, the team will use a dedicated server for all scheduled upload jobs and processes
The team will collaborate internally between engineering, cloud operations, services, and support to monitor all future upload activity, which will only occur during off-business hours.
RCA ISSUE ID: WF-35666

Keywords Waterfall: RCA - 2025/03/25 - UK List Upload Issue

Solution ID

250502082409800

Last Modified Date

05/02/2025 08:32:19 AM

Renew Date

05/02/2026

Status

Published

Author

amuhaz@uplandsoftware.com

Taxonomy

Collections

Upland RightAnswers Portal - Version 2024R2