Data Processing Lagging

Incident Report for MachineMetrics

Resolved

This incident has been resolved.

Posted Jan 25, 2022 - 21:02 EST

Update

The impact of this incident is limited to Automatic Job Dispatch for some machines. The creation of Job Runs based on program data is being processed, but is not operating in real-time. An update will be applied this evening to address performance bottlenecks.

Posted Jan 25, 2022 - 15:37 EST

Identified

The issue has been identified and a fix is being implemented.

Posted Jan 25, 2022 - 12:47 EST

Investigating

Data ingest for a small fraction of machines is behind. The data for the impacted machines will be buffered and processed without data loss once the problem is resolved.

Posted Jan 25, 2022 - 09:38 EST

This incident affected: Web Application, Data Processing Services, and Operator Dashboard.