Evaluating and Scoring Agents
Turn "it seems better" into numbers you can defend: error analysis, code graders, calibrated LLM judges and honest statistics.
2 modules · 6 lessons~16 hoursUpdated 2026-10-30
- Lessons6
- Quizzes6
- Sources38
- Of study~16h
About this course
About this course
An agent you can't measure is an agent you can't safely change. This course teaches the full evaluation loop for agents built on eve and Claude Code: reading traces to find real failure modes, turning them into a task dataset, grading outcomes with code, grading judgement calls with an LLM judge you have calibrated against an expert's labels, and reporting results with trials, pass^k and confidence intervals. Every lesson is grounded in realistic document-workflow agents and ends with hands-on work on your Claude subscription, with eve agents on OpenRouter or a local model.
About the running example
The examples follow a fictional property-law (conveyancing) firm in England and Wales, whose agents draft pre-contract enquiries and are graded against a solicitor's labels. It's a case study, chosen because its documents are long, structured and high-stakes. You don't need a legal background, and the same patterns apply to contracts, insurance claims, invoices, support tickets or patient records. In the lessons, a matter is one client's case and a solicitor is the lawyer who signs work off.
What you'll learn.
- 1Run an error analysis on agent traces and turn the failure taxonomy into a balanced 30+ task dataset
- 2Write code-based graders in eve (defineEval, tool assertions, schema checks, tool stubs, approval checks) and in Claude Code headless runs
- 3Design an LLM judge for one failure mode and calibrate it against expert labels, reporting TPR, TNR and Cohen's kappa
- 4Measure non-deterministic agents with repeated trials, pass@k and pass^k, and put confidence intervals on every score
- 5Compare two agent versions with a paired analysis and gate merges with an eval suite in CI
2 modules, 6 lessons.
7h 30m of lessons
Module 1 · 3 lessons · 3h 30m
Foundations of evaluation
The vocabulary of agent evals, how to find out what actually goes wrong, and the cheapest graders that check it.
Module 2 · 3 lessons · 4h
Scoring that you can trust
LLM judges for what code can't check, calibrating them against an expert, and the statistics that tell signal from noise.
A full eval suite for your drafting agent
Build an evaluation suite you'd trust to gate changes to your drafting agent, from error analysis to 30+ tasks, code graders, a judge calibrated against your own labels, repeated trials with pass^k and a CI regression gate.