Document Ingestion Pipelines
Turn a case's PDFs, scans, forms and emails into verified, structured, privacy-safe data your agents and search can trust.
2 modules · 8 lessons~16 hoursUpdated 2026-10-30
- Lessons8
- Quizzes8
- Sources47
- Of study~16h
About this course
About this course
Every document agent depends on what ingest gives it: the text of a decades-old scanned contract, the entries in an official record, the answers on a handwritten form. This course covers the ingest layer end to end: what a real case file contains, how text-layer extraction, OCR, layout models and vision LLMs compare (and how to read their benchmarks), the hard content (tables, plans, handwriting), schema-validated extraction with evidence quotes, durable pipelines with fan-out, idempotency and poison-document handling, re-ingestion and batch backfills, PII redaction under data-protection law such as GDPR, and the evals that tell you whether any of it works. Code is Python; exercises run on your Claude subscription.
About the running example
The examples follow a fictional property-law (conveyancing) firm in England and Wales: a property purchase file with scanned leases, title registers and plans, and handwritten property forms. It's a case study, chosen because its documents are long, structured and high-stakes. You don't need a legal background, and the same patterns apply to contracts, insurance claims, invoices, support tickets or patient records. In the lessons, a matter is one client's case and a solicitor is the lawyer who signs work off.
What you'll learn.
- 1Inventory a case's documents by type and origin, and route each page to text-layer extraction, OCR, a layout model or a vision LLM on measured cost and accuracy
- 2Read document-parsing benchmarks critically and build a small benchmark sliced to your own documents
- 3Extract schema-validated records with verified evidence quotes and page references, and handle tables, plans and handwriting with explicit review routes
- 4Design a durable ingest pipeline with per-page fan-out, content-hash idempotency keys, retry classes, poison-document quarantine and progress tracking
- 5Re-ingest incrementally by content hash and pipeline version, and plan batch backfills
- 6Detect and pseudonymise personal data for secondary uses under data-protection law such as GDPR, and measure field-level accuracy and coverage before handing documents to retrieval
2 modules, 8 lessons.
9h 45m of lessons
Module 1 · 4 lessons · 4h 50m
Reading real-world documents
From a bundle of PDFs, scans and emails to verified structured records: what the documents are, how each parsing approach works and fails, the hard content, and extraction with evidence.
Module 2 · 4 lessons · 4h 55m
Running ingest in production
The pipeline around the models: durable fan-out, idempotency and poison documents, re-ingestion and backfills, personal data, and the evals and contract that hand documents to retrieval.
Ingest a case file end to end
Build a durable, idempotent ingest pipeline that turns a synthetic contract pack into verified, pseudonymisable records with page-level evidence, survives crashes and poison documents, re-ingests incrementally, and proves its quality with a stratified eval before handing documents to retrieval.