Verify citations end to end (stretch)

Answer-level checks on top of the labelled query set. Retrieved blocks go to Claude as search_result blocks with citations on (or, for any other model, as JSON citations in the same shape). Code then verifies every citation, a claude -p judge checks each claim against only the text it cites and is calibrated on 30 answers you label, and every failure is attributed to retrieval or to generation.

Back to the exercise

Before you start

Before you start:Python with uv Install Claude Code Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

Each check needs a different kind of evidence, so each has its own tool. Code checks citations: does the source exist among what was retrieved, and is the quoted text really in it? A judge checks faithfulness: do the quoted words say what the claim says? String matching against gold answers checks correctness. The query set's labels then do the attribution: if no answering block was retrieved, the model never had a chance. The decision that matters most is giving the judge only the cited text, so it measures support, not plausibility.

Answers Lesson 4: Measuring retrieval

Step-by-step answer

Stretch 60 min7 steps