Module 3 · Benchmarks you can trust · Lesson 9

Reporting results and reading model cards

What a trustworthy benchmark report contains (uncertainty, trials per task, pass^k, cost per success, infrastructure errors, harness and settings), how to compare agents at matched budgets, and how to read a frontier model launch's benchmark footnotes the way a reviewer would.

70 min 2 videos 3 exercises 8-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv

By the end of this lesson you can

List what a benchmark report must disclose for its numbers to be reproducible and comparable, and generate one from per-trial runs and a run manifest
Compute a task-clustered interval, pass^k, cost per success and the infrastructure error rate, and show how the infrastructure-error policy moves a comparison
Compare configurations at matched budgets and say which sit on the cost-quality frontier
Read a model launch's benchmark table and footnotes, sort each comparison into like-for-like, not like-for-like or unknown, and decide whether it should change your own model choice

A launch page says a new model scores 66.4% on an agentic coding benchmark and its nearest competitor 57.9%. Your own report says the Quillmoor support agent passes 82% of tasks on its high-effort configuration and 73% on low effort. Each of these is one number standing in for hundreds of trials, a harness, a set of settings and a cost. A reader can only trust it if the report says what was run, how often, how noisy the result is and what it cost. This lesson defines what a report needs, works one through from per-trial runs, and then reads a real launch table's footnotes the way a reviewer would.

Reporting results and reading model cards

Reporting results and reading model cards

Reporting results and reading model cards

Sign in to keep reading.

Sign in to keep reading.