Run Hugging Face models on CPU

Some exercises load an open-weight model directly in Python: to check your attention code against PyTorch, count a transformer's parameters, compare a base model with its instruct version, or try decoding strategies. Small models (under about 2B parameters) run fine on a laptop CPU. The catch is the install: by default PyTorch on Linux pulls in gigabytes of NVIDIA libraries, and every model you try stays in a cache until you delete it. This page sets both up properly.

For models you only want to chat with or call through an API, Ollama is simpler. Use this page when the exercise needs the model's weights, logits or layers in your own code.

Before you start

• macOS, Linux or Windows, with Python and uv. Python 3.12 is a safe choice.
• About 1 GB of disk for the packages, plus the models: Qwen3-0.6B takes 1.5 GB, and a 1.5B model about 3 GB.
• 8 GB of RAM handles models up to about 1.5B parameters; 16 GB is comfortable.

1. Create the project and get the right PyTorch

uv init hf-cpu --python 3.12
cd hf-cpu

PyPI hosts CPU-only PyTorch wheels for macOS and Windows, but its Linux wheels are built for NVIDIA GPUs: the current release downloads a 540 MB torch wheel plus about 2.1 GB of CUDA libraries. PyTorch's own CPU index has a 210 MB Linux wheel instead. Add this to pyproject.toml before adding torch:

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
 
[tool.uv.sources]
torch = [
  { index = "pytorch-cpu", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
]

• explicit = true means uv uses this index only for packages that name it, so everything else still comes from PyPI.
• The marker applies the CPU index on Linux and Windows. On macOS uv takes PyPI's wheel, which includes Apple GPU support. The lockfile records each platform's choice, so the project installs the same way on any machine.

If you have an NVIDIA GPU and want to use it, leave this block out on Linux, or follow the uv guide for a CUDA index.

2. Add torch and transformers

uv add torch transformers
uv run python -c "import torch; print(torch.__version__)"

On Linux and Windows the version ends in +cpu, for example 2.15.0+cpu. On macOS it has no suffix. If you see a Linux download of nvidia-… packages scroll past, the index block isn't being picked up; see Troubleshooting.

Apple Silicon

Macs with M-series chips can run models on the GPU through PyTorch's MPS backend, with no extra install. Check it:

uv run python -c "import torch; print(torch.backends.mps.is_available())"

True means you can move a model to "mps" instead of "cpu". For a 0.6B model the difference is small; for 1–2B models MPS is noticeably faster. A few operations aren't implemented for MPS yet; if you hit one, setting PYTORCH_ENABLE_MPS_FALLBACK=1 runs it on the CPU instead.

3. Run a smoke test

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
 
device = "mps" if torch.backends.mps.is_available() else "cpu"
model_id = "Qwen/Qwen3-0.6B"
 
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto").to(device)
 
messages = [{"role": "user", "content": "Name the three primary colours, comma-separated."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(device)
output = model.generate(**inputs, max_new_tokens=20, do_sample=False)
new_tokens = output[0][inputs["input_ids"].shape[1]:]
 
print(f"torch {torch.__version__} on {device}")
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

uv run smoke.py

The first run downloads the model (about 1.5 GB) into the Hugging Face cache; later runs load it from disk in a few seconds. Qwen3 models "think" before answering by default, and enable_thinking=False turns that off so 20 tokens are enough. do_sample=False makes the output repeatable.

4. Know where the models go

Downloads go to ~/.cache/huggingface/hub (on Windows, C:\Users\<you>\.cache\huggingface\hub). Every model and every revision stays there until you delete it, and a few exercises' worth of models adds up fast. The hf CLI comes with transformers, so run it through uv:

uv run hf cache ls
uv run hf cache rm model/Qwen/Qwen3-0.6B
uv run hf cache prune

ls lists cached models with their size, rm deletes one (it asks first; add --dry-run to preview), and prune removes old revisions and half-finished downloads. To keep the cache on another disk, set HF_HOME (for example export HF_HOME=/Volumes/External/huggingface) before running anything.

5. Log in for gated models

Most small open models, including Qwen3, download without an account. Some (Meta's Llama and Google's Gemma families, for example) are gated: you must accept the licence on the model's page in a browser first, using your personal Hugging Face account.

Then log your machine in:

uv run hf auth login
uv run hf auth whoami

hf auth login opens a browser flow and saves a token locally; whoami prints your username. You can also create a read token under Settings → Access Tokens and put it in the project's .env as HF_TOKEN=hf_..., kept out of git. A login also raises the anonymous download rate limit, which removes the "unauthenticated requests" warning.

Check it works

uv run smoke.py

Expected output (the device is cpu on Linux and Windows):

torch 2.15.0 on mps
Red, Blue, Yellow

Troubleshooting

Linux still downloads nvidia-… packages. torch was locked before you added the index block. Run uv lock --upgrade-package torch, then uv sync, and check that uv run python -c "import torch; print(torch.__version__)" ends in +cpu.

KeyError: 'qwen3' or "model type not recognised". Your transformers is too old for the model. Run uv add transformers --upgrade-package transformers.

401 or GatedRepoError when loading a model. It's gated. Accept its licence on the Hub, then run uv run hf auth login. Access requests that need manual approval can take a while.

A warning that "Redirects are currently not supported" on macOS. It's harmless logging from PyTorch; ignore it.

Going further

• Local models with Ollama, for chatting with a model rather than loading its weights
• Run GPU code on Modal, when a model is too big for your laptop