Module 1 · Throughput under limits · Lesson 3
Retries, backoff and overload
Classify every failure a model call can produce, retry only what can succeed, with capped exponential backoff, jitter, deadlines and retry budgets, and avoid the retry storms that turn a ten-second provider blip into a long outage.
By the end of this lesson you can
- Classify model-call failures (429 variants, 529, 5xx, timeouts, mid-stream errors, refusals) into retry, wait, fail over or stop
- Implement capped exponential backoff with full jitter that honours retry-after and an end-to-end deadline
- Compute retry amplification across layers and bound it with a single retry layer and a retry budget
- Explain metastable failures and how retries sustain an overload after its trigger has gone
- Reason about what a retried model call costs and repeats, including tool side effects