Gate prompt changes in CI

gate.py runs the tiered regression suite only when a pull request touches prompts, tools, skills or example pools, compares the result with a stored main-branch baseline recorded on the same backend and model, and fails on any control or safety trial, any regression flip, or a capability drop of more than 10 points. Locally it runs on claude -p; in GitHub Actions it uses the OpenAI SDK with your own OpenRouter key, and it writes a Markdown summary for the pull request.

Back to the exercise

Before you start

Before you start:Install Claude Code Python with uv Get an OpenRouter key Local models with Ollama Run evals in GitHub Actions

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

The gate is the suite from the previous exercise turned into a pass/fail check with three rules and a stored baseline. The decision that matters most is what the baseline is: results for main recorded on the same backend and model as the pull-request run. A comparison across models measures the model, not the prompt change, so the gate refuses to run when the two don't match. The second decision is the CI backend: the gate runs claude -p on your subscription locally, but in CI it calls an OpenAI-compatible endpoint with a key you control and can cap.

Gate prompt changes in CI

Back to the exercise

Gate prompt changes in CI

Before you start

Gate prompt changes in CI

Before you start

Sign in to keep reading the answer.

Sign in to keep reading the answer.