Module 3 · Closing the loop with few labels · Lesson 7
Review queues and the dataset loop
Run the human review that every estimate and eval suite depends on, with routing rules and recorded inclusion probabilities, blind and repeated review, tools you can self-host or build yourself, LLM pre-labels that don't anchor reviewers, and promotion of reviewed items into the offline suite with provenance.
Setup for the exercises:Install Claude CodePython with uvTrace LLM calls with OpenTelemetry and Langfuse
By the end of this lesson you can
- Design a set of review queues with routing rules, recorded inclusion probabilities, reviewers per item, blinding, gold items and a weekly capacity plan
- Implement a review queue in self-hosted or free-tier Langfuse or as a small review app built with Claude Code, and compare current tools on queue features, licence and cost
- Decide when an LLM may pre-label or stand in for a reviewer using the alternative annotator test, and measure the anchoring effect of showing pre-labels
- Promote reviewed production items into an offline eval suite with redaction, deduplication, group-aware split assignment, label provenance and retention rules