Module 3 · Benchmarks you can trust · Lesson 7

Auditing benchmarks for exploits

Agents optimise the grader, not the intent. The recurring flaw classes that let an agent score without solving the task, how to red-team your own benchmark with an adversarial coding agent and patch what it finds, and how to report evidence-backed score bounds instead of a bare pass rate.

75 min 2 videos 3 exercises 8-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv Docker and Docker Compose Run agent tasks with Harbor

By the end of this lesson you can

Classify grader flaws into eight recurring classes and name the design change that closes each one
Run white-box and black-box adversarial audits of your own tasks with a coding agent, confirm each exploit with reward-independent evidence, and patch it
Label runs Evidence Pass, Evidence Fail or Unknown and compute evidence-supported score bounds
Explain why pass@k on a deterministic environment equals a replay exploit's success rate, and how varying the environment breaks the equivalence

In May 2026 a team at Berkeley pointed an automated auditing agent at ten popular agent benchmarks, including SWE-bench Verified, SWE-bench Pro, Terminal-Bench, WebArena and OSWorld. On nine of the ten, its exploits scored near-perfectly "without solving a single task" (Wang et al., 2026). One SWE-bench exploit, reconstructed from a reported incident, is a conftest.py file at the repository root: the harness resets the files touched by the official test patch but not new files, pytest loads conftest.py automatically, and a hook in it rewrites every test outcome to "passed". The tasks were fine. The grading trusted something the agent could write. This lesson turns that attacker's view on your own Quillmoor benchmark before an agent finds the holes for you.

Auditing benchmarks for exploits

Auditing benchmarks for exploits

Auditing benchmarks for exploits

Sign in to keep reading.

Sign in to keep reading.