Add a commercial embedder (stretch)

Voyage 4 Lite and Voyage 4 through OpenRouter, the open-weight Nomic baseline through Ollama, on the same 279 chunks and 40 questions, with per-question hits saved for every run. A paired comparison (bootstrap interval and exact McNemar test) then says whether the gap is bigger than noise, and how many questions it would take to see a 3-point difference.

Back to the exercise

Before you start

Before you start:Python with uv Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

This extends the sweep project, so it reuses its corpus, questions, embedder, quantisation and search modules unchanged. Two things are new: each model declares how it wants queries and documents marked (an input_type for Voyage, text prefixes for Nomic), and every run saves a 0 or 1 per question instead of only an average. The second is the decision that matters, because two systems answering the same 40 questions should be compared question by question; only the questions where they disagree say anything about which is better.

Answers Lesson 1: Embeddings in practice

Step-by-step answer

Stretch 45 min6 steps