Measuring retrieval

Every choice so far, the embedding model, the chunk boundaries, the ef_search setting, was a hypothesis. This lesson is how you test them. Without a labelled query set you are tuning on a handful of questions you happen to remember, and every change that fixes one of them silently breaks two you've forgotten. With one, you can say "structure-aware chunking took recall@20 from 0.78 to 0.89 on 180 matter questions" and know whether to ship.

Evaluation of agents in general (error analysis, code graders, LLM judges, calibration and confidence intervals) is the subject of Evaluating and Scoring Agents. This lesson covers what's specific to retrieval: the query set, the ranking metrics and the grounding checks.