Users are unable to get other users' media
Incident Report for LiveKit

Approximately 8:30pm PT we began maintenance to remove unused data centers from our production configuration. Once the changes had been applied at around 9:30pm PT we started to see issues with cross-region communication involving our global messaging bus.

After investigating we found that errors in the above config changes were causing issues in connectivity between the messaging bus. After reverting the changes we were able to restore connectivity between regions at approximately 11:30pm PT.

We've added a few processes to ensure we don't run into similar issues in the future. First, we're adding additional monitoring/paging to ensure we can identify and catch these issues quickly. Second, we've also added additional process around config changes for our global messaging bus and will ensure that configs applied are correct moving forward.

Posted Oct 04, 2022 - 15:39 PDT

Service has been fully restored. Our apologies for the disruptions. A post-mortem will follow in the next few days.
Posted Oct 03, 2022 - 23:33 PDT
The fix has been implemented and service has been restored
Posted Oct 03, 2022 - 23:11 PDT
When users are connected to different server instances, they are currently unable to receive media from one another. We've identified the issue and are working on a fix.
Posted Oct 03, 2022 - 21:30 PDT
This incident affected: Global Real Time Communication.