Incident summary

Some of our services use Cloudflare DNS to resolve addresses of other services. Today when Cloudflare had an incident with their DNS service, it prevented a portion of services from being able to make new connections. Upon discovery, our team applied a fix to move to an alternative DNS provider. This has impacted the following services:

• New inbound and outbound SIP calls
• Creation of new RoomComposite, TrackComposite, and Track Egresses.
• Creation of Ingress streams

New RTC sessions, as well as existing SIP, Egress, and Ingress sessions were not impacted.

Timeline

• 21:55 UTC - alarms started firing due to higher error rate with CreateSIPParticipant API calls
• 21:55 UTC - other health checks started failing across multiple regions - this is unusual.
• 22:03 UTC - the team discovered DNS resolution failures in the error logs, noticing that is preventing connections from completing successfully
• 22:08 UTC - status page is updated with the incident.
• 22:10 UTC - we’ve root caused the issue to be Cloudflare DNS
• 22:15 UTC - we’ve prepared a patch to switch to an alternative DNS provider
• 22:18 UTC - the patch is applied to our entire fleet
• 22:20 UTC - logs confirm errors have stopped and services are restored

Remediation

LiveKit Cloud is designed to survive single point of failure. It’s highly unexpected that multiple data centers would experience symptoms at the same time. Even though we had a backup DNS endpoint, it was also provided by Cloudflare, which proved to be ineffective during this incident.

We’ll be upgrading our DNS fleet to always use servers from multiple provider, to fully mitigate any failures in the DNS infrastructure.