Module 3 · Running models in production · Lesson 9
Choosing and comparing models
How to read benchmarks critically, compare models on cost per completed task, latency and data constraints, decide between frontier APIs and open-weight models, and keep the decision current as new models ship.
By the end of this lesson you can
- Explain what major benchmarks measure and the main ways benchmark scores mislead (saturation, contamination, leaderboard gaming, harness differences)
- Compute cost per completed task, including the cost of failures, and use it to compare models
- List the non-quality constraints (latency, context, data retention and residency, rate limits) that rule models in or out
- Weigh frontier APIs against open-weight models for a given workload
- Design a repeatable model-selection process built on your own eval