At around 16:35 UTC Jan 6, our cloud provider’s API service went down in US East, while we had high traffic volume in the region. This caused issues with autoscaling, as well as with the cloud provider’s Kubernetes CNI plugin. The outcome was that new API requests to LiveKit Cloud would randomly fail, while already established WebSocket/Signal connections remained unaffected. Cloud Dashboard pages could also fail to load.
Customer traffic was diverted to the next closest regions. Once our cloud provider resolved their API issues, we were able to restore functionality of LiveKit Cloud in the US East region. Service was fully restored by 20:35 UTC Jan 6.