Module 2 · Latency, resilience and capacity · Lesson 7
Load testing and capacity planning
Prove the fleet holds under month-end load by replaying recorded traces against a mock model server, then turn the measurements into a capacity plan with Little's law, utilisation, burstiness, Erlang C and pooling, and decide what changes as customers and cases grow.
By the end of this lesson you can
- Explain why LLM workloads are load tested against a mock model server, and build one that reproduces latency, streaming and rate limits
- Build a workload model from recorded traces and drive it with Locust, including a month-end load shape and TTFT as a custom metric
- Find the knee of the latency-throughput curve and report goodput against per-step targets
- Compute queueing delay from utilisation and burstiness with the M/D/1 and Kingman formulas, and size worker pools with Erlang C and square-root staffing
- Produce a capacity plan for fleet growth that combines tier, batch offload, model mix, pooling and headroom