Wire a two-tier eval gate into CI

A two-tier GitHub Actions gate for the eve enquiries agent. The regression tier runs the always-pass evals strictly with three attempts each and uploads the artifacts on failure; the capability tier runs five trials, pairs them with a stored baseline measured on the same pinned model, and fails only if the paired interval's lower end falls below the agreed tolerance. CI runs on your own OpenRouter key, because chatgpt() can't leave your machine.

Back to the exercise

Before you start

Before you start:Durable agents with eve Get an OpenRouter key Run evals in GitHub Actions

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

The two tiers answer different questions, so they use different rules. The regression tier asks "did we break something that always worked?": any failed attempt blocks the merge, which with --repeat 3 makes it a pass^3 gate. The capability tier asks "did we get measurably worse?": its exit code is ignored, and a small script compares per-task pass rates with a baseline and blocks only when the paired 95% interval says the drop could exceed the tolerance. The decision that matters most is that the baseline records which model it was measured on, and the gate refuses to compare across models.

Answers Lesson 6: Statistics, trials and CI gates

Step-by-step answer

Stretch 60–90 min7 steps