Run evals in GitHub Actions

A CI gate runs your evals on every pull request and fails the check when quality drops, so a prompt or model change can't merge on a hunch. The CI-gate exercises build one in GitHub Actions. This page sets up the plumbing they all share: a workflow, a model key stored as a secret, and an artifact holding the results. The eval logic is the exercise's job; see Statistics and CI for why a gate needs paired comparisons.

Before you start

• A personal GitHub account and a repository you own. Don't use an employer's or client's organisation.
• The GitHub CLI (gh auth login done), and a project that already runs its evals locally with one command, such as a uv project with uv run pytest.
• An OpenRouter key created only for CI, with a low credit limit ($2 is plenty). If a workflow loops or a secret leaks, that limit is the most you can lose.

How much Actions costs depends on the repository:

• Public repositories: standard GitHub-hosted runners are free.
• Private repositories: GitHub Free includes $12 of compute a month and 500 MB of artifact storage. A standard Linux runner costs $0.006 a minute, so that's about 2,000 Linux minutes. Windows costs $0.010 a minute and macOS $0.062, so stay on Linux. Without a payment method on file, jobs are blocked when the allowance runs out, so you can't be billed by accident.

1. Store the key as a repository secret

From the repository folder:

gh secret set OPENROUTER_API_KEY

gh asks for the value, so it never lands in your shell history. In the browser, the same thing is Settings → Secrets and variables → Actions → New repository secret. GitHub masks secret values in logs, but a workflow can still print a transformed version, so never echo one.

Secrets are not passed to workflows triggered by pull requests from forks. On your own repository you open pull requests from branches, so this doesn't affect you, and it means a stranger's fork can't spend your key.

2. Add the workflow

name: evals
 
on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/**"
      - "evals/**"
  workflow_dispatch:
 
permissions:
  contents: read
 
concurrency:
  group: evals-${{ github.ref }}
  cancel-in-progress: true
 
jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.draft == false
    env:
      OPENAI_BASE_URL: https://openrouter.ai/api/v1
      OPENAI_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
      MODEL: anthropic/claude-haiku-5.5
    steps:
      - uses: actions/checkout@v7
 
      - uses: astral-sh/setup-uv@v10.3.3
        with:
          enable-cache: true
 
      - run: uv sync --locked
 
      - name: Run evals
        run: uv run pytest evals --junitxml=results/junit.xml
 
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: eval-results
          path: results/
          retention-days: 7

What each part does:

• pull_request with paths runs the gate only when files that can change model behaviour change, not on every README edit.
• workflow_dispatch adds a Run workflow button, so you can run the suite on demand. It only appears once the file is on the default branch.
• permissions: contents: read gives the job's token read access and nothing more.
• concurrency cancels the previous run when you push again to the same branch, so you pay for one run, not five.
• timeout-minutes: 15 stops a hung model call from burning minutes. The default is 360.
• The if: line skips draft pull requests. Mark a PR ready when you want the gate.
• env follows the OpenRouter pattern: the OpenAI SDK reads OPENAI_BASE_URL and OPENAI_API_KEY itself. Exercises using the Claude Agent SDK set ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN and an empty ANTHROPIC_API_KEY instead.
• upload-artifact with if: always() keeps the results even when the gate fails, which is exactly when you need them. Seven days of retention keeps storage low; the default is 90.

setup-uv only publishes full version tags, so v10.3.3 is the current release at the time of writing. For the other actions, the major tag (v7) follows their latest release.

Swap the Run evals command for whatever your exercise uses, such as uv run python gate.py or pnpm exec eve eval --strict --junit. Keep the results inside results/ so the upload step finds them.

3. Push and run it

git add .github/workflows/evals.yml
git commit -m "Add eval workflow"
git push
gh workflow run evals.yml
gh run watch

gh workflow run needs the workflow on the default branch, so push to main first (or merge it there through a pull request). gh run watch follows the latest run live.

4. Read the results

gh run download --name eval-results --dir eval-results

In the browser, the artifact appears at the bottom of the run's Summary page.

Keep usage down

• Run the cheap, deterministic tests on every push and the model-calling evals only through this gated workflow.
• Use a small, cheap model in CI, and a fixed slug rather than a latest alias so results are comparable.
• Check usage under Settings → Billing and licensing on your personal account. With a payment method on file, set a budget for Actions there so usage beyond the included amount stops instead of billing you.
• Check spend per run on the CI key's OpenRouter Activity page.

Check it works

gh run list --workflow evals.yml --limit 1

Expected: one run, triggered by workflow_dispatch, with status completed and conclusion success (or failure if your gate failed on purpose), and gh run download producing eval-results/junit.xml.

Troubleshooting

HTTP 422: Workflow does not have 'workflow_dispatch' trigger, or no Run workflow button. The workflow isn't on the default branch yet. Merge it to main.

The model call fails with HTTP 401 in CI but works locally. The secret name in the workflow doesn't match the one you set (gh secret list shows the names), or the run came from a fork.

The run waits in "Queued" or says the job was not started. Your private repository's included compute is used up. Wait for the monthly reset, make the repository public if it holds nothing private, or add a payment method and a budget.

uv sync --locked fails. uv.lock is missing or out of date. Run uv lock locally and commit the lockfile.

Going further

• Get an OpenRouter key, for the CI key and its credit limit
• Python with uv