Agent Benchmarks and Simulations
Evaluate multi-turn, stateful agents with simulated users and real environments, and build benchmarks whose numbers survive an audit.
3 modules · 9 lessons~32 hoursUpdated 2026-11-01
- Lessons9
- Quizzes9
- Sources58
- Of study~32h
About this course
About this course
Single-task evals stop working once an agent holds a conversation and changes the world it acts on. This course builds the next layer: stateful environments graded on the end state they are left in, user simulators that talk back and must be validated before you trust them, tasks packaged for current harnesses such as Harbor and Inspect, and the infrastructure that keeps scores reproducible. The second half turns you into a benchmark auditor: contamination, saturation and broken tasks, exploits that score without solving, item analysis and time horizons, and reports and model cards read the way a reviewer would. Code is Python with Docker, and agents run as Claude Code on your Claude subscription.
About the running example
The examples follow Quillmoor, a fictional company that sells project-management software, and its account-support agent. The agent changes plans and seats, issues refunds within a written policy (duplicate charges included) and recovers accounts through tools over a SQLite backend, while simulated customers (some impatient, some uncooperative) talk to it. Every account, invoice and conversation is synthetic.
What you'll learn.
- 1Build a stateful evaluation environment (backend as tools, a written policy, gold-action replay) and grade agents on end state, collateral damage and required communication
- 2Write a user simulator with a persona, a hidden goal, a stop condition, its own tools and uncooperative variants, and measure its fidelity and its sensitivity to the simulator model
- 3Package tasks for a current harness (Harbor or Inspect) with sandboxes and oracle and empty-agent controls, and run repeated trials of Claude Code reproducibly from a run manifest
- 4Audit a benchmark for contamination, saturation, broken or underspecified tasks and exploitable graders, and fix what you find
- 5Fit item difficulty and discrimination to a trials matrix, prune a suite without losing ranking fidelity, and explain how time-horizon metrics are computed and where they break
- 6Report results with a noise breakdown, cost per success and reliability metrics, and critique a frontier model's benchmark claims from its footnotes
3 modules, 9 lessons.
11h 20m of lessons
Module 1 · 3 lessons · 3h 40m
Simulated worlds
Replace fixtures with a world that changes and a user who talks back, then check that the simulated user behaves like a real one.
Module 2 · 2 lessons · 2h 35m
Running benchmarks
Turn tasks into a harness you and others can rerun, and separate the noise from the signal.
Module 3 · 4 lessons · 5h 5m
Benchmarks you can trust
Find out whether a number means what it claims, close the exploits that inflate it, and report it honestly.
A benchmark for the account-support agent you'd defend
Build a 24-task benchmark for a stateful, conversational agent, with simulated customers, Harbor packaging, controls and repeated trials from a manifest, then audit it for broken tasks and exploits, analyse it item by item, and publish a report card whose numbers survive a reviewer.
More in Evaluation and Optimisation.
- IntermediateAI agents · Evals
Evaluating and Scoring Agents
Turn "it seems better" into numbers you can defend: error analysis, code graders, calibrated LLM judges and honest statistics.
7 lessons ~20hFree - AdvancedEvals · Human feedback
Human Feedback at Scale
Collect, measure and trust the judgements of reviewers and users: study design, rater agreement, preference rankings, real-use signals and quality estimates from a few labels.
7 lessons ~28hFree - AdvancedFine-tuning · Reinforcement learning
Fine-Tuning and RL for Agents
Train a small open model for one agent step with supervised fine-tuning, preference optimisation and reinforcement learning, and prove with your eval suite whether it beats an optimised prompt.
8 lessons ~38hFree