Module 3 · Closing the loop with few labels · Lesson 6

Estimating quality from few labels

Spend a small human-labelling budget where it counts, combine a judge's verdicts on all traffic with those labels to get an unbiased quality estimate with an honest interval, and know how many labels the judge can really save.

80 min 1 video 3 exercises 8-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv

By the end of this lesson you can

Design stratified and importance-weighted review samples and compute unbiased estimates from them with inverse-probability weights
Compute a prediction-powered estimate and 95% interval from judge verdicts on all items plus human labels on a random subset, and compare it with human-only and Rogan–Gladen estimates
Explain power tuning (PPI++) and compute how many human-only labels a judge-assisted sample is worth from the judge–human correlation
Explain when uncertainty-driven (active) sampling reduces the labels needed, and why it must be reweighted
Use the factor-of-two limit to set a realistic labelling budget when the judge is no more accurate than the system it grades

Every Monday, Quillmoor's quality report needs one number: the share of last week's 5,000 or so drafted support replies that the senior support lead would approve. The LLM judge you calibrated in Calibrating judges scores all of them for almost nothing, but it was calibrated weeks ago on a different mix of tickets. The lead can label about 200 a week, and a pass rate from 200 labels comes with an interval of about ±6 points, too wide to see a 3-point regression. This lesson is about getting more out of those 200 labels: how to choose them, how to combine them with the judge into one estimate that is unbiased however good or bad the judge is, and how much the judge can save at best.

Estimating quality from few labels

Estimating quality from few labels

Estimating quality from few labels

Sign in to keep reading.

Sign in to keep reading.