Scaling AI Systems
Keep a fleet of agents fast and available under real load: rate limits, flow control, retries, backpressure, caching, failover, latency budgets, load tests and capacity plans.
2 modules · 7 lessons~18 hoursUpdated 2026-10-30
- Lessons7
- Quizzes7
- Sources44
- Of study~18h
About this course
About this course
What changes when one agent becomes thousands of model calls a minute across many customers and cases. The first module works inside a provider's limits: how RPM, ITPM and OTPM are enforced and read, client-side concurrency and token-bucket limiters, retries that don't turn a blip into an outage, and queues that absorb the month-end rush. The second module makes the fleet fast and sturdy: latency budgets per agent step, response and semantic caching (and where semantic caching is unsafe for case-specific work), failover across Claude on Anthropic, Bedrock and Google Cloud, and load testing against a mock model server, ending in a capacity plan built on queueing theory. Every number is recomputed in Python, every claim is tied to docs or papers, and every lesson has exercises you run with Claude.
About the running example
The examples follow a fictional property-law (conveyancing) firm in England and Wales and the other firms on the same platform, including the month-end rush of completions. It's a case study, chosen because its documents are long, structured and high-stakes. You don't need a legal background, and the same patterns apply to contracts, insurance claims, invoices, support tickets or patient records. In the lessons, a matter is one client's case and a solicitor is the lawyer who signs work off.
What you'll learn.
- 1Compute which provider limit (RPM, ITPM or OTPM) binds a workload, including the effect of prompt caching, and read the rate-limit headers to prove it
- 2Implement token-aware client-side limiters and concurrency control in async Python, shared across workers and fair across customers
- 3Design retry policies with jitter, retry budgets and deadlines that recover from 429 and 529 errors without amplifying an overload
- 4Absorb load spikes with priority queues, admission control, load shedding and batch processing, and compute backlog and drain times
- 5Set a latency budget per agent step and meet it with streaming, parallel tool calls, prompt caching and prefetching, and decide where response or semantic caching is safe
- 6Load test an agent fleet against a mock model server with replayed traces, and produce a capacity plan from Little's law, utilisation and tail-latency targets
2 modules, 7 lessons.
8h 55m of lessons
Module 1 · 4 lessons · 4h 55m
Throughput under limits
The provider's limits, the client-side controls that keep you inside them, the retry policies that survive errors, and the queues that absorb spikes.
Module 2 · 3 lessons · 4h
Latency, resilience and capacity
Make each agent step fast, serve some answers without the model, survive a provider's bad day, and prove the whole fleet holds up under month-end load.
Month-end readiness for an agent fleet
Build and prove the dispatch layer for three deadline-day agents serving five synthetic customers, against a mock Claude API on two platforms. A token-aware fair limiter, a retry policy with budgets and breakers, a priority admission queue with a batch path, latency and caching work, warm-standby failover, and a month-end load test that ends in a capacity plan.
More in Production AI at Scale.
- AdvancedAI agents · Infrastructure
Infrastructure for Agents
Run a fleet of agents in production: compute, sandboxes, durable workflows, queues, state, tenant isolation, gateways and safe delivery.
8 lessons ~16hFree - AdvancedAI agents · Security
Production Agent Systems
Take an agent from working demo to a system you can run for months: architecture, security, compliance, observability and safe change.
6 lessons ~18hFree