Module 2 · Latency, resilience and capacity · Lesson 5
Latency engineering for agents
Where an agent turn's seconds go (queueing, time to first token, thinking, generation and tools), how to set a latency budget per step, and the levers that move each term, from streaming and parallel tool calls to cache pre-warming, prefetching and speculation.
Setup for the exercises:Install Claude CodePython with uvGet an OpenRouter keyLocal models with Ollama
By the end of this lesson you can
- Decompose the latency of a model call and of an agent turn into queueing, TTFT, thinking, generation and tool time
- Measure TTFT, time to first visible text and output speed from a stream, and report percentiles rather than means
- Set a latency budget per agent step from a user-facing target and identify which term dominates
- Apply streaming, parallel tool calls, cache pre-warming, prefetching and output control, and compute what each saves
- Decide when speculative execution or hedged requests are worth their cost in tokens and rate-limit capacity