Global Analytics Dashboard was offline
Incident Report for LiveKit
Postmortem

Summary

Between the hour of 12:00PM EST and 12:20PM EST on Tuesday, 31st May 2022 users encountered Global Analytics Dashboard throwing 500 error.

The event was triggered by a Analytics Database reconfiguration at 12:00PM EST.

Root Cause

The Analytics database reconfiguration was to reduce the number of background workers based on the DB configuration which required a restart of the database

The shutdown of the analytics database took longer than expected, which lead to the analytics database being offline.

This critical incident affect 100% of Analytics dashboard users.

Corrective actions

  1. Schedule Maintenance window to have least amount of impact
  2. Publish maintenance window in advance
Posted May 31, 2022 - 10:15 PDT

Resolved
A unscheduled analytics database reconfiguration led to the database taking longer than expected to reboot. This lead to analytics dashboard being offline for 20 minutes, during which only the analytics was impacted.
Posted May 31, 2022 - 09:00 PDT