Contamination, saturation and broken tasks

On 23 February 2026 OpenAI stopped reporting SWE-bench Verified, the benchmark it had built 18 months earlier and that every frontier launch had quoted since. Scores had crept from 74.9% to 80.9% in six months, and the question was whether the remaining failures said anything about the models. An audit said mostly not: most of the hard remaining tasks had tests that reject correct fixes, and the models could recite some of the original fixes from memory (OpenAI, 2026). The number was still being computed correctly. It had stopped meaning what people said it meant. This lesson is about noticing that before a launch post does it for you, in public benchmarks and in the Quillmoor suite you built in the Simulated worlds and Running benchmarks modules.

LLMs for Engineers › Choosing and comparing models covers the reader's side: how a published score misleads someone choosing a model. Here you are the producer and auditor, and you measure those problems instead of listing them.