Build and measure a hybrid pipeline

BM25 with an identifier-safe analyser, dense retrieval through the OpenAI SDK, RRF and a local cross-encoder over the top 100, plus an optional claude -p listwise rerank whose IDs are validated in code, all measured per stratum on 60 labelled queries over a synthetic corpus of 24 matters and a precedent library.

Back to the exercise

Before you start

Before you start:Install Claude Code Python with uv Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

Every configuration shares one pipeline that records each stage's ranking and time for each query, so BM25, dense, RRF, RRF plus cross-encoder and the LLM stage are all scored on exactly the same queries in one pass. The decision that matters most is the analyser: title numbers, planning references and clause numbers are matched as whole tokens before anything else, because the corpus is built so that those identifiers are the only thing separating one matter from its neighbour. Dense retrieval goes through the OpenAI SDK (OpenRouter or Ollama). The dry run uses a deterministic fake embedder, and the cross-encoder in "Run it" is the real local model.

Answers Lesson 5: Hybrid search and reranking

Step-by-step answer

Core 90 min8 steps