High API failure rate due to load balancer failure
Incident Report for LiveKit

At around 16:35 UTC Jan 6, our cloud provider’s API service went down in US East, while we had high traffic volume in the region. This caused issues with autoscaling, as well as with the cloud provider’s Kubernetes CNI plugin. The outcome was that new API requests to LiveKit Cloud would randomly fail, while already established WebSocket/Signal connections remained unaffected. Cloud Dashboard pages could also fail to load.

Customer traffic was diverted to the next closest regions. Once our cloud provider resolved their API issues, we were able to restore functionality of LiveKit Cloud in the US East region. Service was fully restored by 20:35 UTC Jan 6.

Posted Jan 06, 2023 - 12:52 PST

This incident has been resolved.
Posted Jan 06, 2023 - 12:34 PST
A fix has been implemented and we are monitoring the results.
Posted Jan 06, 2023 - 12:29 PST
We are continuing to work on a fix for this issue.
Posted Jan 06, 2023 - 09:30 PST
We've identified an issue with our cloud provider that is causing random failures with API requests in our US East region. We are continuing to monitor the situation.
Posted Jan 06, 2023 - 09:00 PST
This incident affected: Cloud Dashboard (cloud.livekit.io).