Designing human evaluations

In Calibrating judges, one principal expert labelled a gold set and the judge was measured against her. That works while the question is "does this output meet our standard?" and one person's standard is the standard. Quillmoor's support assistant now drafts replies for 14 support agents and answers about 20,000 help-centre conversations a month, and the questions have changed: are prompt v5's drafts better than v4's for the agents who send them, which kinds of ticket does the assistant fail, and does the judge still agree with people this quarter? The support lead can't label her way through that alone, and some of those questions aren't hers to answer. Anthropic's guide to building evaluations sums up human grading as "Most flexible and high quality, but slow and expensive. Avoid if possible." This lesson is about the cases where you can't avoid it, and how to design a human study that is cheap enough to run and good enough to trust.