17:45 UTC New deployment of cloud dashboard service was triggered. The configuration contained decommissioned database endpoints, which the new service tried to establish connections unsuccessfully, and subsequent pings on the failed connection led to panics and service going offline

‌

18:15 UTC we identified the decommissioned database endpoints and updated the configurations, and the service was back online

‌

Remediation steps:

1. Add checks to prevent pings on failed database connections
2. Read/sync configuration from central configuration repository to prevent reading stale configurations