Run evals with Inspect

Inspect is an open-source evaluation framework from the UK AI Security Institute and Meridian Labs. An evaluation is a Task made of a dataset of samples, a solver that produces an answer (one model call or a full agent), and a scorer, optionally with a sandbox per sample for tools that run code. Exercises use it to evaluate models through an API with sandboxed tools and inspectable logs. Inspect calls models through providers with an API key, not through your Claude subscription, so you'll need your own OpenRouter key or a local Ollama model. Task specs, harnesses and sandboxes compares it with Harbor and τ-bench.

Before you start

• uv and Python 3.12 or newer (Python with uv). Inspect itself supports Python 3.11+.
• A model: an OpenRouter key with a credit limit set on it, or Ollama running with a model that can call tools (see OpenRouter or a local model?).
• Docker for sandboxed tasks (Docker and Docker Compose). Inspect checks for Docker Engine 24.0.6 or newer and Docker Compose 2.21.0 or newer.

1. Create a project

uv init inspect-sandbox && cd inspect-sandbox
uv add inspect-ai openai

The openai package is the client Inspect uses for both the openrouter/ and the ollama/ providers.

2. Configure a model in .env

Inspect reads a .env file from the folder you run it in (and from its parents). Create one:

INSPECT_EVAL_MODEL=openrouter/anthropic/claude-haiku-5.5
OPENROUTER_API_KEY=sk-or-...

For a free local model instead:

ollama pull qwen3.5:9b

INSPECT_EVAL_MODEL=ollama/qwen3.5:9b
# OLLAMA_BASE_URL=http://localhost:11434/v1   (the default)

Add .env to .gitignore before you commit anything. INSPECT_EVAL_MODEL is the default for --model. You can override it on any run with --model openrouter/anthropic/claude-sonnet-5.5 or any other provider-prefixed name.

3. Write and run a first task

from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate
 
 
@task
def hello():
    return Task(
        dataset=[Sample(input="Reply with the single word: ready", target="ready")],
        solver=generate(),
        scorer=includes(),
    )

uv run inspect eval hello.py

Inspect shows progress while it runs and writes a log to ./logs/.

4. Add a Docker sandbox

Tools that run commands, such as bash(), need a sandbox. Each sample gets its own container, and files copies inputs into it:

from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.tool import bash
 
 
@task
def find_code():
    return Task(
        dataset=[Sample(
            input="A file in the current directory contains a four-digit code. Find it and submit it.",
            target="4417",
            files={"note.txt": "The code is 4417."},
        )],
        solver=react(tools=[bash()]),
        scorer=includes(),
        sandbox="docker",
    )

uv run inspect eval sandboxed.py

With sandbox="docker" and no config files, Inspect uses its standard tool-support image, and the compose file it generates switches networking off. If you add your own compose.yaml next to the task, it replaces that generated file, so put network_mode: none in it yourself unless the task needs the network.

5. Read the logs

uv run inspect view

It serves the logs in ./logs/ on a local URL. Open a sample to see the messages, every tool call, the score and the scorer's explanation.

Options you'll use

Check it works

Run the first task from Python and print its status and accuracy:

uv run python -c "from inspect_ai import eval; log = eval('hello.py')[0]; print(log.status, log.results.scores[0].metrics['accuracy'].value)"

Expected output, after Inspect's progress display:

success 1.0

Troubleshooting

No API key or an authentication error. The provider's variable is missing in the folder you ran from: check that .env is in the project folder and spells OPENROUTER_API_KEY correctly. For Ollama, check that ollama serve is running and the model is pulled (ollama list).

An import error naming openai. The provider needs the openai package: uv add openai.

Docker errors when a sandboxed task starts. Make sure Docker is running, then check versions with docker version and docker compose version against the minimums above. With many samples at once on a laptop, lower --max-sandboxes.

Samples fail with rate-limit errors, or one error stops the whole eval. Lower --max-connections. Use --retry-on-error 2 for transient errors, and --no-fail-on-error to keep the other samples running, then look at the errored samples in inspect view.

Going further

• Run agent tasks with Harbor, to run CLI agents such as Claude Code inside task containers on your subscription
• Inspect's agent bridge runs agents such as Claude Code in a sandbox, routing their model calls through Inspect's provider
• Noise, reproducibility and eval infrastructure, on epochs, limits, retries and what they do to a score