An eval-gated model swap in eve CI

An eve agent whose model can be swapped for eval runs only, twelve tagged evals run three times each with --strict and --junit, a script that pairs two runs' .eve/evals artifacts by eval and by tag and applies pre-written blocking rules, and a GitHub Actions workflow that rehearses the whole pipeline with a mock model and runs the real comparison on your OpenRouter key.

Back to the exercise

Before you start

Before you start:Install Claude Code Durable agents with eve Get an OpenRouter key Local models with Ollama Run evals in GitHub Actions

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

eve eval already does the hard parts: it boots the agent, runs each eval several times and writes every attempt to .eve/evals/<timestamp>/. What it doesn't do is compare two runs. So the upgrade path is: run the evals on the current model, save the artifacts, run them on the candidate, save those, and let a small script pair them and decide. The decision that matters most is where the candidate model comes from: an environment variable that eval runs set and production can never honour, so trying a model can't quietly become shipping it.

Answers Lesson 6: Continuous improvement

Step-by-step answer

Stretch 90 min8 steps