Runbooks
Surge mode engagement
SEV3 runbook for surge mode auto-engaging on traffic spike.
Last updated
Surge mode engagement (SEV3)
Triggered by: traffic ≥ 5× baseline for ≥ 5 minutes
(apps/api/lib/surge-mode.ts).
What surge mode does
- Engages load-shed automatically.
- Increases per-token rate-limit headroom temporarily (controlled burst absorption).
- Raises cache TTL on idempotent reads to deflect repeat work.
- Dispatches the surge-mode notification to status page.
Diagnose
Typically benign — surge mode is the system absorbing traffic gracefully. Verify:
matter ops surge-statusInvestigate the surge source:
- Customer-side: legitimate growth, marketing campaign, integration.
- Adversarial: cross-check with
ddos-response.mdx.
Recover
Surge mode exits automatically when traffic normalises for 10 minutes.
Post-engagement
- If customer-side surge, reach out to the customer about capacity planning.
- If chronic, plan a capacity adjustment.