Outage Causes

Trigger

1. We started receiving a rapid spike of StartEgress in one of our clusters. The request rate was 10x of what we typically receive

   

2. Our infrastructure is setup with some amount of idle egress instances to absorb extra requests. This amount was tuned for the typical egress request pattern.

Fallout

1. The egress pods with extra capacity started accepting the new requests. The autoscaler started bringing up new pods as needed to keep the idle pod count on target. The rate of new pod creation was thus limited by the target idle pod count. The pods were allocated from a pool of warm nodes available to absorb request bursts in either the Egress or cloud-media clusters.
2. After 4 min the infrastructure ran out of warm nodes and started provisioning new nodes from our cloud provider. Node provisioning takes several minutes and was only done a few nodes at a time because of the aforementioned scaling algorithm.
3. The Egress API started returning 503 errors to a percentage of new API requests. Existing egresses were not affected.
4. After 25 min, the cluster had started enough new pods to absorb the peak in Egress requests. This was the end of the customer impact

Contributing factors

1. The alert threshold for API error rate for Egress were misconfigured, delaying notification and escalation of the issue.
2. The size of the warn node pools was too small for this cluster.

Remediation

1. The algorithm used to compute the target amount of idle Egress pods has been modified to keep a significantly larger buffer, in order to be able to absorb greater traffic spikes.
2. Alerts threshold were changed to provide a quicker escalation of 503 errors in the Egress API.
3. We are working on adjusting the size of the warm node pool.