Module 1 · Spending less on the same work · Lesson 2

Prompt caching as system design

Turning a per-request cache into a fleet-wide strategy, with layered prefixes, cache-aware scheduling, the arithmetic of cache lifetimes, and hit rate as a monitored objective.

75 min 1 video 3 exercises 6-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv Get an OpenRouter key

By the end of this lesson you can

Derive the break-even number of reads for a cache write, and use it to choose between 5-minute and 1-hour lifetimes
Estimate hit probability from request arrival rates, and explain why per-case prefixes and shared prefixes need different strategies
Design a layered prompt layout and breakpoint plan for a fleet of agents
Schedule work so caches are warm when parallel or follow-up requests arrive
Monitor cache hit rate per agent and diagnose drops

The LLMs for Engineers lesson on caching covered the mechanism: an exact prefix match over tools, system and messages, cheap reads, slightly dearer writes, a 5-minute or 1-hour lifetime. That is enough to make one agent cache well. A fleet is a different problem. Twelve agents touch the same matter at different times, with different tools, sometimes in parallel, sometimes after a fee earner has been away for twenty minutes. Whether the cache hits is decided less by any single prompt than by how you lay out prompts across agents and when you send them.

Prompt caching as system design

Turning a per-request cache into a fleet-wide strategy, with layered prefixes, cache-aware scheduling, the arithmetic of cache lifetimes, and hit rate as a monitored objective.

Prompt caching as system design

Prompt caching as system design

Sign in to keep reading.

Sign in to keep reading.