Measure truncation and quantisation on legal text

A sweep over full, 256 and 64 dimensions at float32, int8 and binary precision, with and without 4x rescoring, on 279 synthetic conveyancing chunks and 40 labelled questions. Vectors come once from any OpenAI-compatible embeddings endpoint and are cached, so every configuration is computed from the same embeddings and only the compression changes.

Back to the exercise

Before you start

Before you start:Python with uv Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

Embed the corpus once at full width and full precision, cache it, and derive every other configuration from those vectors in numpy: truncate and re-normalise for the dimension, then quantise for the precision. That keeps the comparison fair and cheap, because the model is called twice (documents, then questions) and fifteen configurations come out. The decision that matters most is calibrating int8 on the documents only and applying those same ranges to queries, so both live on one scale.

Answers Lesson 1: Embeddings in practice

Step-by-step answer

Core 60–90 min8 steps