Resolved -
This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.
Aug 19, 17:30 PDT
Monitoring -
Mitigations have been deployed and we are now catching up to realtime.
Aug 19, 17:01 PDT
Update -
The amount of data we are ingesting in the sessions view is outpacing our ability to ingest, thus causing a delay in ingestion. We are pursuing a few mitigations right now including moving to a larger database instance to relieve the processing bottleneck.
Aug 19, 15:07 PDT
Update -
We are continuing to work on a fix for this issue.
Aug 19, 14:25 PDT
Identified -
Sessions data ingest remains delayed for about an hour. It's not impacting other dashboards.
We've identified the source of the slow ingest and are working on a mitigation. We will update again in 30 mins.
Aug 19, 14:22 PDT
Investigating -
We are currently investigating delayed session data ingestion on Cloud Dashboard. No services appear to be impacted and we don't expect any data to be lost.
Aug 19, 12:33 PDT