Module 2 · Scoring that you can trust · Lesson 5

Calibrating judges

Measure how far you can trust an LLM judge by comparing it with a domain expert's labels, using TPR, TNR and Cohen's kappa, then iterate the prompt and correct the rates you report.

75 min 2 videos 3 exercises 7-question quiz Answers

Setup for the exercises:Install Claude Code Node.js, pnpm and TypeScript Durable agents with eve

By the end of this lesson you can

Build a labelled gold set with a principal domain expert, split into train, dev and test sets
Compute a judge's TPR, TNR, raw agreement and Cohen's kappa from a confusion matrix, and explain why raw agreement misleads
Diagnose judge disagreements and iterate the judge prompt without leaking test examples
Correct an observed pass rate for judge error, and know when that correction breaks down
Choose a threshold for eve's t.judge scores from labelled data, and decide when to recalibrate

Your redundancy judge from the last lesson looks sensible on the five enquiries you tried. Would you let it decide, on your behalf, whether this week's prompt change made the enquiries agent worse? Not yet. A judge is a classifier, and nobody ships a classifier without measuring its error rates on labelled data. This lesson does exactly that: a solicitor labels a gold set, you compare the judge with those labels, and you only trust the judge's numbers once you know how often, and in which direction, it's wrong.

Calibrating judges

Measure how far you can trust an LLM judge by comparing it with a domain expert's labels, using TPR, TNR and Cohen's kappa, then iterate the prompt and correct the rates you report.

Calibrating judges

Calibrating judges

Sign in to keep reading.

Sign in to keep reading.