Module 3 · Running models in production · Lesson 7

Context windows, KV cache and caching

What happens to your prompt at inference time, why long context costs memory and money, how the KV cache and prompt caching work, and why a million-token window is not a million tokens of attention.

75 min 2 videos 3 exercises 6-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv Run Hugging Face models on CPU

By the end of this lesson you can

Distinguish prefill from decode and relate each to time-to-first-token and output speed
Compute the KV-cache size of a model from its config, and explain how grouped-query attention shrinks it
Explain prompt caching as reuse of a prefix's KV cache, and compute the savings for a realistic workload
Structure a prompt so the cache hits, and diagnose the common silent invalidators
Summarise the evidence that model quality degrades with long or cluttered contexts

Frontier models now accept a million tokens of input. That number hides three different costs: GPU memory to hold the context while generating, money and latency to process it, and a quality cost, because models do not use every token of a long context equally well. This lesson works through all three, starting with the mechanics of inference and ending with prompt caching, the single biggest lever on the cost of an agent that keeps re-reading the same documents.

Context windows, KV cache and caching

What happens to your prompt at inference time, why long context costs memory and money, how the KV cache and prompt caching work, and why a million-token window is not a million tokens of attention.

Context windows, KV cache and caching

Context windows, KV cache and caching

Sign in to keep reading.

Sign in to keep reading.