Global Analytics Dashboard was offline

Incident Report for LiveKit

Postmortem

Summary

Between the hour of 12:00PM EST and 12:20PM EST on Tuesday, 31st May 2022 users encountered Global Analytics Dashboard throwing 500 error.

The event was triggered by a Analytics Database reconfiguration at 12:00PM EST.

Root Cause

The Analytics database reconfiguration was to reduce the number of background workers based on the DB configuration which required a restart of the database

The shutdown of the analytics database took longer than expected, which lead to the analytics database being offline.

This critical incident affect 100% of Analytics dashboard users.

Corrective actions

Schedule Maintenance window to have least amount of impact
Publish maintenance window in advance

Posted May 31, 2022 - 10:15 PDT

Resolved

A unscheduled analytics database reconfiguration led to the database taking longer than expected to reboot. This lead to analytics dashboard being offline for 20 minutes, during which only the analytics was impacted.

Posted May 31, 2022 - 09:00 PDT