Module 2 · Running benchmarks · Lesson 5
Noise, reproducibility and eval infrastructure
Where the variance in an agent benchmark score comes from beyond the model (task sampling, resource limits, time limits, concurrency and provider errors), and how to run large eval jobs that are isolated, pinned, logged and comparable.
By the end of this lesson you can
- Decompose a score's variance into prediction noise and data noise, estimate both from a trials matrix, and say whether more trials or more tasks reduces each
- Separate infrastructure errors from agent failures with a classification policy fixed before the run, and treat resource and time limits as controlled variables
- Explain how dropping or zero-filling errored trials biases a score, and compute a less biased one
- Write a run manifest that pins the benchmark, harness, agent version, model, effort, limits, concurrency and seeds
- Run trials in parallel without shared state, cache only what doesn't change the measurement, and keep full traces for review