Human Feedback at Scale
Collect, measure and trust the judgements of reviewers and users: study design, rater agreement, preference rankings, real-use signals and quality estimates from a few labels.
3 modules · 7 lessons~28 hoursUpdated 2026-11-01
- Lessons7
- Quizzes7
- Sources47
- Of study~28h
About this course
About this course
Judges and code graders only stay honest while people keep checking them, and one expert stops being enough once an agent serves thousands of users. This course teaches the human side of evaluation: designing human studies and guidelines, measuring agreement among many raters, ranking versions from pairwise preferences, turning thumbs, edits and retries into signals you have validated, mapping what thousands of conversations are about without reading them, and combining a judge on all traffic with a small human sample to get honest quality estimates. Code is Python with free open-source tools, every dataset is synthetic with known ground truth, and exercises run on your Claude subscription.
About the running example
The examples follow Quillmoor, a fictional mid-size company that sells project-management software. Its support assistant drafts replies to customer tickets for fourteen support agents to approve or edit, and answers questions in the help-centre chat, about 20,000 conversations a month. All tickets, ratings and feedback events are synthetic, generated with planted ground truth so every estimate can be checked against the truth.
What you'll learn.
- 1Design a human evaluation (question format, rater pool, guideline, pilot and budget) and report it so someone else could reproduce it
- 2Measure the reliability of many raters with Krippendorff's alpha and a bootstrap interval, diagnose disagreement, and choose how to aggregate labels
- 3Rank several system versions from pairwise votes with a Bradley–Terry model, confidence intervals and style controls, and critique a public leaderboard
- 4Instrument user feedback linked to traces, measure each signal's validity against expert labels, and correct a feedback-based rate for selection bias
- 5Build a privacy-preserving map of what conversations are about, with facets, clusters and aggregation thresholds, and validate it on synthetic data
- 6Estimate production quality from a judge on all traffic plus a small human sample with prediction-powered inference, and run the review queues that feed the offline suite
3 modules, 7 lessons.
8h 50m of lessons
Module 1 · 3 lessons · 3h 45m
Human judgement at scale
When one principal expert is no longer enough: design human studies, measure many raters, and rank versions from preferences.
Module 2 · 2 lessons · 2h 30m
Signals from real use
What users tell you without being asked, and what thousands of conversations are about, measured honestly and privately.
Module 3 · 2 lessons · 2h 35m
Closing the loop with few labels
Spend human labels where they count, combine them with judges to get honest numbers, and run the queues that turn reviewed traffic into the next offline suite.
Build a feedback system for a support assistant
Design and run the human side of evaluation for an assistant that drafts support replies, on a synthetic week with planted ground truth. A guideline and a multi-rater pilot, a pairwise ranking of three prompt versions, user signals you have validated, a private map of what conversations are about, a weekly quality estimate from a judge plus a few human labels, the review queue that feeds it all, and a one-page measurement plan.
More in Evaluation and Optimisation.
- IntermediateAI agents · Evals
Evaluating and Scoring Agents
Turn "it seems better" into numbers you can defend: error analysis, code graders, calibrated LLM judges and honest statistics.
7 lessons ~20hFree - AdvancedAI agents · Evals
Agent Benchmarks and Simulations
Evaluate multi-turn, stateful agents with simulated users and real environments, and build benchmarks whose numbers survive an audit.
9 lessons ~32hFree - AdvancedPrompt engineering · DSPy and GEPA
Prompt and Context Optimisation
Treat prompts, examples, tool descriptions and skills as versioned, measured artefacts, and improve them by hand, by optimiser and in production without breaking what already works.
7 lessons ~17hFree