Cloud Agents deploy issues

Incident Report for LiveKit

Postmortem

This issue was caused by lock contention in the cloud agents deployment code path. This caused some builds to not get deployed in a timely manner.

The offending lock scope has been decreased significantly which should ensure this issue doesn’t happen again. We’ve also added additional monitoring around the queue involved to ensure we are notified earlier of any similar issues.

Posted Sep 10, 2025 - 12:32 PDT

Resolved

This incident has been resolved.

Posted Sep 10, 2025 - 12:27 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Sep 10, 2025 - 11:37 PDT

Identified

The issue has been identified and a fix is being implemented.

Posted Sep 10, 2025 - 10:47 PDT

Investigating

Some cloud agent builds are having problems getting scheduled; we're investigating this issue.

Posted Sep 10, 2025 - 10:09 PDT

This incident affected: Global Cloud Agents.