Build a skill-trigger eval in Claude Code

A skill-trigger eval on your subscription. Each query runs through claude -p in a throwaway project that holds only the skill, the stream-json events are scanned for the Skill tool call, and precision, recall and F1 are computed per trial and per query. v2 of the description lifts dev F1 from 0.67 to 0.90, and the held-out set shows 0.75, which is the gap you have to explain.

Back to the exercise

Before you start

Before you start:Install Claude Code Python with uv

The eval is a binary classifier test: 40 labelled queries, three claude -p runs each, and one yes/no per run saying whether the model loaded raising-leasehold-enquiries. The decision that matters most is where each run happens: a fresh temporary project containing only this skill, with the description under test written into its frontmatter. Your other skills, CLAUDE.md files and the rest of this folder can't steal a query or cause a trigger, and swapping descriptions is a file write, not a hand edit.

Build a skill-trigger eval in Claude Code

Back to the exercise

Build a skill-trigger eval in Claude Code

Before you start

Build a skill-trigger eval in Claude Code

Before you start

Sign in to keep reading the answer.

Sign in to keep reading the answer.