At exactly 00:00 UTC, our wildcard TURN TLS certificate expired, which lead to certain network-restricted clients being unable to connect to our RTC backend. Even though we anticipated the certificate expiring and already had a new certificate available, the certificate had not been applied to our fleet. While we do have monitoring in place for our TURN TLS service, due to a limitation of our monitoring software, we were not monitoring the TLS certificate.
We were made aware of a connectivity issue after receiving a customer report at 00:09 UTC. We were able to deduce the root cause of the issue by 00:40 UTC, and we quickly reconfigured our TURN TLS servers with the updated TLS certificate. RTC connectivity via TURN TLS was restored at 00:48 UTC.
Going forward, we will implement a process to ensure accountability in manual configuration changes, such as updating our TURN TLS certificate. We also have a plan to automate this process in the future, and implement additional monitoring.
Note that TURN UDP was unaffected throughout this outage.