Transcript-driven tool improvement

A four-tool MCP server over synthetic contract packs, built with the MCP Python SDK 2.x in two versions (v1 mediocre, v2 after review). 16 code-checked tasks run through claude -p with only that server attached, the dev transcripts are fed to a separate claude -p proposer, and the approved changes lift task success from 3/10 to 9/10 on dev and 2/6 to 5/6 on held-out.

Back to the exercise

Before you start

Before you start:Install Claude Code Python with uv MCP servers in Claude Code

The workflow is the one from Anthropic's tools post, made reproducible: run, read the transcripts, change the tools, re-run on tasks the changes never saw. The decision that matters most is that the two tool sets live side by side in the server (TOOLSET=v1 or v2) and the system prompt never changes, so every difference in the table is caused by tool names, descriptions, parameters and error messages. The proposer only ever sees dev transcripts, and the held-out run refuses to start until you say the tools are frozen.

Transcript-driven tool improvement

Back to the exercise

Transcript-driven tool improvement

Before you start

Transcript-driven tool improvement

Before you start

Sign in to keep reading the answer.

Sign in to keep reading the answer.