Module 2 · Running ingest in production · Lesson 5
Durable ingest pipelines
The architecture around the models for long-running ingest jobs - units of work and fan-out per page and document, content-derived idempotency keys, retry classes, poison-document quarantine and progress tracking that stays true under retries.
By the end of this lesson you can
- Choose the unit of work and retry for each ingest stage, and compute the rework cost of retrying at the wrong granularity
- Derive idempotency keys from content hashes and stage versions, and make every stage an idempotent upsert
- Classify ingest failures into transient, deterministic and quality failures, and handle each differently, including poison documents
- Implement a case ingest as a durable workflow with per-document child workflows and per-page activities in Python
- Derive progress from artefacts rather than counters, and explain why counters drift