Measure flakiness and compare two prompt versions

Sixteen stubbed eve tasks run 5 times each on a baseline and a candidate prompt, with a stats.ts (per-task rates, pass@1, unbiased pass^3 and pass@3, naive and task-clustered standard errors, Wilson) and compare.ts (paired per-task differences with a normal and a bootstrap interval). The candidate looks 11 points better, and the paired interval says that isn't yet evidence.

Back to the exercise

Before you start

Before you start:Durable agents with eve Get an OpenRouter key Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

Repeated trials only measure something if the only thing that varies between them is the model. So every task serves the same synthetic pack through tool stubs, the graders are deterministic code, and the two prompt versions run on the same 16 tasks with the same number of trials. The statistics then treat the task as the unit: per-task pass rates first, then means, standard errors and differences over tasks. The decision that matters most is pairing: comparing per-task differences cancels task difficulty, which here varies far more than the prompts do.

Measure flakiness and compare two prompt versions

Back to the exercise

Measure flakiness and compare two prompt versions

Before you start

Measure flakiness and compare two prompt versions

Before you start

Sign in to keep reading the answer.

Sign in to keep reading the answer.