Module 2 · Memory in production · Lesson 8
Evaluating memory
How to tell whether memory helps. Ground-truth-first test cases, ablations against no memory, full history and an oracle, metrics for staleness, abstention, leaks and poisoning, and the full cost and latency of writing and retrieving memories.
By the end of this lesson you can
- Build a memory test set from facts with validity intervals before any text is generated, with questions at several points in a case
- Run ablations (no memory, full history, memory system, oracle) and attribute failures to the write, retrieval or reading stage
- Measure stale answers, false memories, cross-case leaks and poisoning resistance, and bound a zero-failure leak rate with the rule of three
- Compute the per-case cost of a memory design, including write-time extraction and cold caches between sessions, and compare it with full history
- Decide from evidence whether to ship, tune or remove a memory component