Expired TURN TLS Certificate
Incident Report for LiveKit
At exactly 00:00 UTC, our wildcard TURN TLS certificate expired, which lead to certain network-restricted clients being unable to connect to our RTC backend. Even though we anticipated the certificate expiring and already had a new certificate available, the certificate had not been applied to our fleet. While we do have monitoring in place for our TURN TLS service, due to a limitation of our monitoring software, we were not monitoring the TLS certificate.

We were made aware of a connectivity issue after receiving a customer report at 00:09 UTC. We were able to deduce the root cause of the issue by 00:40 UTC, and we quickly reconfigured our TURN TLS servers with the updated TLS certificate. RTC connectivity via TURN TLS was restored at 00:48 UTC.

Going forward, we will implement a process to ensure accountability in manual configuration changes, such as updating our TURN TLS certificate. We also have a plan to automate this process in the future, and implement additional monitoring.

Note that TURN UDP was unaffected throughout this outage.
Posted Oct 19, 2023 - 17:00 PDT