Run agent tasks with Harbor

Harbor runs agent tasks in containers. A task is a directory: an instruction, an environment, a test script that writes a reward, and a reference solution. Harbor runs an agent on each task in a fresh sandbox, grades the result and keeps every trajectory. It's the official harness for Terminal-Bench, and the exercises in Agent Benchmarks and Simulations use it to package their own tasks and to run Claude Code on your subscription. The ideas behind the task format are in Task specs, harnesses and sandboxes.

Before you start

• Docker running. Harbor uses Docker as its default sandbox. Follow Docker and Docker Compose first and check that docker run --rm hello-world works.
• uv, from Python with uv. Harbor needs Python 3.12 or newer, which uv installs for you.
• Claude Code signed in with your own Pro, Max or Ultra plan (Install Claude Code). You only need it for step 4.
• Disk and time. Each task builds an image, and agent trials install the agent in the container. The first run of a task takes minutes, later runs reuse the cached layers. Keep 10 GB free.

1. Install Harbor

Install Harbor as a uv tool, so the harbor command is available everywhere without touching your projects:

uv tool install harbor
harbor --help

To upgrade to the current release later, reinstall it:

uv tool uninstall harbor
uv tool install harbor

If you don't use uv, pip install harbor works too.

2. Run the oracle on a sample task

The oracle agent runs a task's reference solution instead of a model, so it costs nothing and tells you that Docker, the task and the verifier all work:

mkdir harbor-sandbox && cd harbor-sandbox
harbor run -t hello-world/hello-world -a oracle

Harbor pulls the task, builds its image, runs the solution, runs the tests, and writes everything under ./jobs/. Each trial gets its own folder with a result.json (rewards and any exception), the agent's logs and the verifier's output. Open the local viewer to browse them:

harbor view jobs

It prints a local URL (the first free port from 8080). Keep the command running while you look.

The other built-in control is nop, an agent that does nothing. On a sound task it scores 0:

harbor run -t hello-world/hello-world -a nop

3. Scaffold your own task

harbor task init my-task

This creates instruction.md, task.toml, environment/Dockerfile, solution/solve.sh and tests/. The test script must write a number to /logs/verifier/reward.txt (or labelled numbers to /logs/verifier/reward.json). New tasks are set up with a separate verifier: tests run in a fresh container, and only files you list as artifacts in task.toml are copied across from the agent's container.

Check the task with both controls, and lint it:

harbor run -p my-task -a oracle   # must score 1
harbor run -p my-task -a nop      # must score 0
harbor lint my-task

-p also accepts a folder of task directories, which Harbor runs as a local dataset.

4. Run Claude Code on your subscription

Harbor's claude-code agent installs Claude Code inside the task container. To bill your Claude subscription rather than an API key, give it a long-lived OAuth token. Create one on your machine:

claude setup-token

It opens the same browser sign-in as /login and prints a token valid for one year. It doesn't save the token anywhere. Export it in the shell you run Harbor from:

export CLAUDE_CODE_OAUTH_TOKEN=your-token

Then run the agent, with three independent attempts and two trials at a time:

harbor run -p my-task -a claude-code -m anthropic/claude-sonnet-5-5 \
  --ae CLAUDE_FORCE_OAUTH=1 -k 3 -n 2

--ae sets an environment variable for the agent. CLAUDE_FORCE_OAUTH=1 tells Harbor's Claude Code integration to drop any API key it finds, so the run can't fall back to API billing. Without it, Harbor uses ANTHROPIC_API_KEY whenever both are set.

Each attempt is a full Claude Code session and counts against your plan's usage limits, so start with a few tasks and -k 3.

Flags you'll use

harbor run --help lists the rest, and the full list of agents.

Check it works

harbor run -t hello-world/hello-world -a oracle --job-name check && \
  find jobs/check -mindepth 2 -maxdepth 2 -name result.json | xargs cat

Expected: one trial's JSON, in which verifier_result.rewards shows a reward of 1 and exception_info is null.

Troubleshooting

harbor: command not found after installing. uv's tool folder isn't on your PATH. Run uv tool update-shell, then open a new terminal.

Cannot connect to the Docker daemon, or the environment never starts. Docker Desktop or OrbStack isn't running. Start it and wait until it says it's running; on Linux run sudo systemctl start docker. A trial that fails here is recorded as a sandbox error, not as a failed attempt.

Claude Code trials fail on authentication, or usage shows up on an API account. Check that echo $CLAUDE_CODE_OAUTH_TOKEN prints the token in the shell you run harbor from, and pass --ae CLAUDE_FORCE_OAUTH=1. Unset ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN and ANTHROPIC_BASE_URL if you don't need them.

The first agent run is slow. Harbor builds the image and installs the agent before the first attempt. Install the agent's dependencies in your Dockerfile (Claude Code needs curl), and keep -n low on a laptop so trials don't compete for CPU and memory.

Going further

• Run evals with Inspect, for model evals through an API with sandboxed tools
• MCP servers in Claude Code: Harbor registers servers declared under [[environment.mcp_servers]] in task.toml for agents that support them
• Simulate a user: --user-agent, --user-model and --bridge acp run a second agent that holds the instruction and talks the agent under test through the task
• Noise, reproducibility and eval infrastructure, on what concurrency, retries and resource limits do to the results