Module 2 · Running ingest in production · Lesson 8
Ingest evals and the hand-off
How to know whether ingest works - page-level unit tests, field-level accuracy, precision, recall, hallucination rate and coverage on a stratified gold set, online health checks - and the contract that hands documents to retrieval.
By the end of this lesson you can
- Build a stratified, pseudonymised gold set and write page-level unit tests in the style of olmOCR-Bench
- Compute field-level accuracy, precision, recall, hallucination rate on empty fields and coverage, with confidence intervals
- Explain why aggregate accuracy can hide invented values, and weight fields by severity
- Design online health checks and drift alerts that run on every document without labels
- Specify the hand-off contract and events that feed the retrieval system, without doing retrieval's job