Compare single-shot, grep-only and full agentic retrieval

One synthetic leasehold matter with block IDs, 30 labelled questions (18 multi-hop or absence), and three strategies run through claude -p on your subscription, single-shot hybrid RAG, Claude Code with Read, Grep and Glob, and the same plus a search command, each logged as stream-json and scored on evidence recall over the trajectory, correctness, citations, tool calls, tokens and time.

Back to the exercise

Before you start

Before you start:Install Claude Code Python with uv Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

All three strategies answer with the same rules and the same JSON schema, and every run is logged with --output-format stream-json, so the only thing that changes between them is how the model gets to the text. The decision that matters most is measuring evidence recall over the trajectory: every block ID that appears in the prompt or in any tool result counts as seen, and an answer that is correct with no gold block seen is counted as a lucky guess, not a success. The hybrid retriever's dense half goes through the OpenAI SDK (OpenRouter or Ollama), and a fake embedder is used when none is configured.

Answers Lesson 6: Agentic retrieval

Step-by-step answer

Core 2 hours7 steps