Local models with Ollama

Ollama runs open-weight models on your own computer and serves them on http://localhost:11434. It speaks two APIs: a subset of the Anthropic Messages API, which Claude Agent SDK and Anthropic SDK code uses, and an OpenAI-compatible API, which OpenAI-SDK code uses for chat, structured output and embeddings. Exercise code reaches either by changing environment variables, not code. Everything stays on your machine and costs nothing per token.

Be realistic about quality. A model that fits on a laptop is far weaker than Claude at multi-step tool use, long documents and following a strict output format. Local models are good for learning how the plumbing works, for cheap iteration and for private data. Expect some agent exercises to work worse, and switch to a strong model through OpenRouter when you need the real result.

Before you start

• macOS 14 Sonoma or newer. Apple silicon runs models on the GPU; Intel Macs run them on the CPU only.
• Windows 10 22H2 or newer. NVIDIA (driver 551.61+) and AMD GPUs are supported.
• Linux with an NVIDIA or AMD GPU for usable speed. CPU-only works but is slow.
• Memory is the constraint. The model's download size plus its context has to fit in VRAM, or in unified memory on a Mac. A rough rule: download size plus a few GB of headroom.
• Disk: 1 to 20 GB per model.

1. Install Ollama

macOS and Linux

curl -fsSL https://ollama.com/install.sh | sh

On macOS you can instead download the app from ollama.com/download and drag it to Applications.

Windows

irm https://ollama.com/install.ps1 | iex

Or run OllamaSetup.exe from the download page. It installs in your user folder and doesn't need Administrator rights.

On macOS and Windows the app keeps a server running in the background. On Linux the install script sets up a service; if it isn't running, start it with ollama serve. Check with:

ollama -v

2. Pick models that fit your machine

Sizes are the default download from the Ollama library. All the chat models listed support tool calling.

New models arrive every few weeks. The library's tools filter lists current models that can call tools, with their sizes. Answers default to qwen3.5 for chat and embeddinggemma for embeddings, and read both from environment variables so you can swap them.

3. Pull and run a model

ollama pull qwen3.5
ollama run qwen3.5 "Explain a context window in two sentences."
ollama pull embeddinggemma

ollama run qwen3.5 with no prompt opens a chat; type /bye to leave. Other commands you'll use:

ollama list          # downloaded models
ollama ps            # loaded models, memory use, GPU/CPU split, context size
ollama stop qwen3.5  # unload from memory
ollama rm qwen3.5    # delete from disk

4. Raise the context length for agents

Ollama sizes the context by VRAM: 4K tokens below 24 GB, 32K from 24 to 48 GB, and 256K from 48 GB up. A 4K window is too small for agents and long documents, and the model silently loses earlier text. Ollama recommends at least 64K for agents and coding tools:

• App (macOS, Windows): Settings, then drag the context length slider.
• CLI: quit the app first, then run

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Check the CONTEXT and PROCESSOR columns of ollama ps. If the processor shows part CPU, the model plus context doesn't fit in VRAM; use a smaller model or a shorter context.

5. Use the Anthropic-compatible API

Claude Agent SDK and Anthropic SDK code reaches Ollama through these variables in the project's .env. The token is required but ignored, and the empty ANTHROPIC_API_KEY stops a real key from taking over:

ANTHROPIC_BASE_URL=http://localhost:11434
ANTHROPIC_AUTH_TOKEN=ollama
ANTHROPIC_API_KEY=
MODEL=qwen3.5

curl http://localhost:11434/v1/messages \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{"model":"qwen3.5","max_tokens":200,"messages":[{"role":"user","content":"Hello!"}]}'

Messages, streaming, system prompts, tools and tool results, base64 images and thinking all work. These parts of the Anthropic API don't:

Partly supported: images must be base64 (no URLs), and the thinking budget_tokens value is accepted but not enforced. Structured-output (JSON schema) settings aren't in this endpoint's supported fields list either, so answers validate JSON themselves; see OpenRouter or a local model?.

6. Use the OpenAI-compatible API and embeddings

The same server exposes OpenAI-style endpoints under /v1. OpenAI-SDK code (plain model calls, structured output, embeddings) reaches them through the two variables the SDK reads itself, so the client is created with no arguments. The key is a placeholder that Ollama ignores:

OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=ollama
MODEL=qwen3.5
EMBEDDINGS_MODEL=embeddinggemma

curl http://localhost:11434/v1/embeddings \
  -H "content-type: application/json" \
  -d '{"model":"embeddinggemma","input":["first text","second text"]}'

/v1/chat/completions supports tools, JSON mode and response_format, but not tool_choice, logprobs or image URLs (send images as base64).

7. Point Claude Code at a local model

Ollama can start Claude Code with the right settings:

ollama launch claude

Or set the variables yourself for one session. ANTHROPIC_API_KEY="" stops a key in your environment from taking over:

ANTHROPIC_AUTH_TOKEN=ollama ANTHROPIC_BASE_URL=http://localhost:11434 ANTHROPIC_API_KEY="" claude --model qwen3.5

Run /status inside the session to confirm the base URL. Set the variables inline like this, not in your shell profile, or every claude -p call in your exercises will go to Ollama too. Claude Code's system prompt and tool definitions are large, so use a model with at least 64K context. Even then, small models often stall in multi-step edits.

LM Studio is an alternative desktop app that downloads and serves local models on OpenAI-compatible endpoints (by default at http://localhost:1234/v1).

Check it works

curl -s http://localhost:11434/v1/messages \
  -H "content-type: application/json" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"qwen3.5","max_tokens":50,"messages":[{"role":"user","content":"Reply with the single word: ready"}]}' \
  | jq -r '.content[] | select(.type=="text") | .text'

Expected output (thinking models may add a thinking block first, which the filter skips):

ready

Troubleshooting

connection refused on port 11434. The server isn't running. Open the Ollama app, or run ollama serve on Linux.

ollama serve fails with "address already in use". The app's server is already running. Quit the app before running ollama serve with your own settings.

A 404 or "model not found" error. Run ollama pull qwen3.5 first. Model names must match ollama list exactly, including the tag.

Very slow answers, or the model forgets the start of the conversation. Check ollama ps. Part-CPU means the model doesn't fit in memory; pick a smaller one. A 4096 context means you haven't raised it (step 4).

Going further

• OpenRouter or a local model?, the configuration pattern every answer uses
• Get an OpenRouter key, for any hosted model
• Install Claude Code