Choose a model for document extraction

A complete harness for the capstone. Thirty synthetic registers with gold records, a token profile, a streaming claude -p runner that logs usage, cache, thinking, TTFT and cost with retries and timeouts, a scorer with critical fields, evidence quotes checked against the cited entry, a caching experiment, a models × effort sweep run twice, and the analysis and memo that turn it into a decision.

Back to the project

The harness separates three things that are easy to blur: what the model said (one JSONL row per call, every attempt kept), whether it was right (a scorer and a quote check that are pure functions of the output and the source), and what it cost (usage, cache fields, thinking tokens, latency and API-equivalent cost from Claude Code's own result). Everything runs through claude -p on your Claude subscription. The decision that matters most is fixing the scorer's normalisation rules, the critical fields and the failure cost before the first sweep, and running every configuration at least twice, so that the final choice is made against run-to-run noise rather than inside it. Every document in data/ is synthetic: the people, lenders, addresses and title numbers are invented.

Back to the project

Choose a model for document extraction

Choose a model for document extraction

Sign in to keep reading the answer.

Sign in to keep reading the answer.