Module 2 · Scoring that you can trust · Lesson 6

Statistics, trials and CI gates

Separate signal from noise in agent evals with repeated trials, pass@k and pass^k, confidence intervals and paired comparisons, then gate merges in CI and keep watching in production.

90 min 1 video 3 exercises 8-question quiz Answers

Setup for the exercises:Durable agents with eve Get an OpenRouter key Local models with Ollama Run evals in GitHub Actions

By the end of this lesson you can

Explain where variance in agent eval scores comes from, and run repeated trials with eve's --repeat and Claude Code headless
Compute pass@k and pass^k from trial counts, including the unbiased estimators, and choose the right one for a given agent
Put a confidence interval on a pass rate, cluster it by task when tasks have several trials, and estimate how many tasks you need
Compare two agent versions with a paired analysis and say whether the difference is real
Design a two-tier CI gate and a production monitoring loop with traces and a human review queue

Prompt v2 of the enquiries agent scores 82% on your 40-task suite. v1 scored 78%. Do you merge? You can't tell yet. Run v1 again and it might score 75% or 83%, because the agent samples its outputs, the judge samples its verdicts, and 40 tasks are a small sample of the files the agent will meet. This lesson gives you the tools to answer "is v2 better?" with a number and an interval, and then to make that answer automatic in CI.

Statistics, trials and CI gates

Separate signal from noise in agent evals with repeated trials, pass@k and pass^k, confidence intervals and paired comparisons, then gate merges in CI and keep watching in production.

Statistics, trials and CI gates

Statistics, trials and CI gates

Sign in to keep reading.

Sign in to keep reading.