Build and run a labelled query set

An 80-question set over one synthetic matter and a 50-clause precedent library, in five strata, with graded block-level qrels built by pooling BM25 and dense retrieval. A runner reports recall@10, MRR@10 and nDCG@10 per stratum for both retrievers with bootstrap intervals over queries, and a claude -p judge, run twice, is checked against your labels on 40 pooled pairs.

Back to the exercise

Before you start

Before you start:Python with uv Install Claude Code Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

The query set is a test suite, so it lives in the repository as two small JSON files: queries.json (text, stratum, source) and qrels.json (block id to grade). Everything else is code that reads them. Retrieval here works on blocks rather than chunks, so a retrieved item is already the unit that was labelled. The decision that matters most is pooling before labelling: you label the union of two different retrievers' top 20, so neither retriever is graded only on what it already finds.

Answers Lesson 4: Measuring retrieval

Step-by-step answer

Core 90–120 min7 steps