Evaluating and Scoring Agents
Turn "it seems better" into numbers you can defend: error analysis, code graders, calibrated LLM judges and honest statistics.
2 modules · 7 lessons~20 hoursUpdated 2026-11-02
- Lessons7
- Quizzes7
- Sources45
- Of study~20h
About this course
About this course
An agent you can't measure is an agent you can't safely change. This course teaches the full evaluation loop for agents built on eve and Claude Code: reading traces to find real failure modes, turning them into a task dataset, grading outcomes with code, grading judgement calls with an LLM judge you have calibrated against an expert's labels, and reporting results with trials, pass^k and confidence intervals. Every lesson is grounded in realistic document-workflow agents and ends with hands-on work on your Claude subscription, with eve agents on OpenRouter or a local model.
About the running example
The examples follow a fictional property-law (conveyancing) firm in England and Wales, whose agents draft pre-contract enquiries and are graded against a solicitor's labels. It's a case study, chosen because its documents are long, structured and high-stakes. You don't need a legal background, and the same patterns apply to contracts, insurance claims, invoices, support tickets or patient records. In the lessons, a matter is one client's case and a solicitor is the lawyer who signs work off.
What you'll learn.
- 1Run an error analysis on agent traces and turn the failure taxonomy into a balanced 30+ task dataset
- 2Write code-based graders in eve (defineEval, tool assertions, schema checks, tool stubs, approval checks) and in Claude Code headless runs
- 3Design an LLM judge for one failure mode or a weighted rubric for each task, and calibrate it against expert labels, reporting TPR, TNR and Cohen's kappa
- 4Measure non-deterministic agents with repeated trials, pass@k and pass^k, and put confidence intervals on every score
- 5Compare two agent versions with a paired analysis and gate merges with an eval suite in CI
2 modules, 7 lessons.
8h 45m of lessons
Module 1 · 3 lessons · 3h 30m
Foundations of evaluation
The vocabulary of agent evals, how to find out what actually goes wrong, and the cheapest graders that check it.
Module 2 · 4 lessons · 5h 15m
Scoring that you can trust
LLM judges and per-task rubrics for what code can't check, calibrating them against an expert, and the statistics that tell signal from noise.
A full eval suite for your drafting agent
Build an evaluation suite you'd trust to gate changes to your drafting agent, from error analysis to 30+ tasks, code graders, a judge calibrated against your own labels, repeated trials with pass^k and a CI regression gate.
More in Evaluation and Optimisation.
- AdvancedPrompt engineering · DSPy and GEPA
Prompt and Context Optimisation
Treat prompts, examples, tool descriptions and skills as versioned, measured artefacts, and improve them by hand, by optimiser and in production without breaking what already works.
7 lessons ~17hFree - AdvancedEvals · Human feedback
Human Feedback at Scale
Collect, measure and trust the judgements of reviewers and users: study design, rater agreement, preference rankings, real-use signals and quality estimates from a few labels.
7 lessons ~28hFree - AdvancedFine-tuning · Reinforcement learning
Fine-Tuning and RL for Agents
Train a small open model for one agent step with supervised fine-tuning, preference optimisation and reinforcement learning, and prove with your eval suite whether it beats an optimised prompt.
8 lessons ~38hFree