Add an open-weight model to the bake-off

The model bake-off extended with an OpenAI-SDK adapter that sends the same schema and prompt to open-weight models on OpenRouter (DeepSeek V4.1 Flash, MIT; Qwen3.8 27B, Apache 2.0) or to a local Ollama model, records latency, tokens and OpenRouter's reported cost, and appends the rows to the same table and plot.

Back to the exercise

Before you start

Before you start:Install Claude Code Python with uv Get an OpenRouter key OpenRouter or a local model?Local models with Ollama

This exercise calls a model. Run it either way

Any model through OpenRouterOne key for Claude, GPT, Gemini, Llama and more. Pay per token, with a credit limit.A local open-source modelFree and private with Ollama. Weaker on hard tasks.

Which should I pick? →

The bake-off already holds everything constant except the model; this answer adds a second door into the same harness. The adapter uses the OpenAI SDK created with no arguments, so OPENAI_BASE_URL and OPENAI_API_KEY decide whether a request goes to OpenRouter or to a local Ollama model, and its results come back in the same shape as a claude -p row. The decision that matters most is keeping the task identical: the same system prompt, the same user prompt and the same JSON Schema, sent as response_format: json_schema with strict: true, falling back to plain JSON mode only for a provider that rejects it, and recording which mode was used.

Answers Lesson 9: Choosing and comparing models

Step-by-step answer

Stretch 60 min7 steps