Serve a model with vLLM

vLLM is a widely used open-source server for running open-weight models in production. It batches many requests onto one GPU and exposes an OpenAI-compatible API, so code written for OpenRouter or Ollama talks to it unchanged. Exercises on serving, throughput and inference cost use it. vLLM needs an NVIDIA GPU on Linux; if you don't have one, rent one by the second on Modal.

Before you start

• On Modal (recommended without an NVIDIA GPU): Modal set up with a budget. The server below runs on an L4, about $0.80 an hour while a container is up.
• On your own GPU: Linux (or WSL on Windows; vLLM doesn't run on Windows natively), an NVIDIA GPU with compute capability 7.5 or newer (T4, RTX 20-series and later) and about 16 GB of VRAM for the model below, plus uv.
• A Mac, or a laptop without a big GPU, is better served by Ollama.

The model is Qwen3-4B-Instruct-2507: Apache 2.0, ungated (no Hugging Face token needed), 8 GB of weights, and an instruct model without a separate thinking mode, so replies are plain answers.

1. Create an API key

A vLLM server on a public URL answers anyone who finds it, so require a key. Generate one and keep it:

export VLLM_API_KEY=$(openssl rand -hex 24)
echo "$VLLM_API_KEY"

vLLM reads VLLM_API_KEY (or --api-key) and rejects requests without it.

2. Option A: serve on Modal

Store the key as a Modal Secret, so the container gets it as an environment variable:

uv run modal secret create vllm-api-key VLLM_API_KEY="$VLLM_API_KEY"

This is Modal's official vLLM example with four changes for learning: a small model, a cheap L4 instead of an H200, an API key, and limits so it never runs more than one GPU or idles for long.

import modal
 
MODEL_NAME = "Qwen/Qwen3-4B-Instruct-2507"
MODEL_REVISION = "cdbee75f17c01a7cc42f958dc650907174af0554"  # pin the weights
MINUTES = 60  # seconds
 
vllm_image = (
    modal.Image.from_registry("nvidia/cuda:12.9.0-devel-ubuntu22.04", add_python="3.12")
    .entrypoint([])
    .uv_pip_install("vllm==0.21.0")  # the version Modal's example pins
    .env({"HF_XET_HIGH_PERFORMANCE": "1"})
)
 
hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True)
vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True)
 
app = modal.App("vllm-qwen")
 
 
@app.server(
    image=vllm_image,
    gpu="L4",
    secrets=[modal.Secret.from_name("vllm-api-key")],  # sets VLLM_API_KEY
    volumes={
        "/root/.cache/huggingface": hf_cache_vol,
        "/root/.cache/vllm": vllm_cache_vol,
    },
    port=8000,
    startup_timeout=10 * MINUTES,
    scaledown_window=5 * MINUTES,  # scale to zero after 5 idle minutes
    max_containers=1,  # never run more than one GPU
    unauthenticated=True,  # vLLM checks the API key instead
)
class Server:
    @modal.enter()
    def start(self):
        import subprocess
 
        cmd = [
            "vllm", "serve", MODEL_NAME,
            "--revision", MODEL_REVISION,
            "--served-model-name", MODEL_NAME,
            "--host", "0.0.0.0",
            "--port", "8000",
            "--max-model-len", "16384",
            "--enforce-eager",  # faster cold starts, slightly slower tokens
        ]
        self.process = subprocess.Popen(cmd)
 
    @modal.exit()
    def stop(self):
        self.process.terminate()

The Volumes cache the weights and vLLM's compiled kernels, so only the first start downloads 8 GB. Deploy it:

uv run modal deploy vllm_server.py

The command prints the server URL, shaped like https://<your-workspace>--vllm-qwen-server.us-east.modal.direct. A server with no containers answers 503 to every request while it starts one, so wake it and wait for the health check:

export VLLM_URL=https://<your-workspace>--vllm-qwen-server.us-east.modal.direct
until curl -sf "$VLLM_URL/health"; do sleep 5; done; echo ready

The first start takes several minutes (image build plus download); later cold starts are faster because the image and weights are cached.

3. Option B: serve on your own NVIDIA GPU

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen3-4B-Instruct-2507 --max-model-len 16384

--torch-backend=auto picks the PyTorch build for your CUDA driver. The server reads VLLM_API_KEY from the shell you exported it in and listens on http://localhost:8000. The official vllm/vllm-openai Docker image is the alternative if you'd rather not install it; see the GPU installation guide.

4. Call it with the OpenAI SDK

vLLM plugs into the shared configuration pattern as one more .env block. MODEL is the Hugging Face name the server was started with:

Modal:

OPENAI_BASE_URL=https://<your-workspace>--vllm-qwen-server.us-east.modal.direct/v1
OPENAI_API_KEY=<your VLLM_API_KEY>
MODEL=Qwen/Qwen3-4B-Instruct-2507

Local GPU:

OPENAI_BASE_URL=http://localhost:8000/v1
OPENAI_API_KEY=<your VLLM_API_KEY>
MODEL=Qwen/Qwen3-4B-Instruct-2507

Keep .env in .gitignore. Any answer's OpenAI-SDK code then runs unchanged, for example model.py from OpenRouter or a local model?. One server serves one model, so point EMBEDDINGS_MODEL at OpenRouter or Ollama if an exercise also embeds.

Check it works

curl -s "$OPENAI_BASE_URL/models" -H "Authorization: Bearer $OPENAI_API_KEY"

Expected: JSON whose data list contains "id": "Qwen/Qwen3-4B-Instruct-2507". Without the header you get a 401, which shows the key is enforced.

When to use Ollama instead

Use Ollama for everyday local development on a laptop. Use vLLM when an exercise is about serving: concurrency, throughput, latency under load, or what production inference looks like.

Troubleshooting

503 Service Unavailable from Modal. No container is up yet. Run the /health loop above and retry once it prints ready.

CUDA out of memory at startup. The model plus its cache doesn't fit. Lower --max-model-len (for example 8192), or pick a smaller model such as Qwen/Qwen3-1.7B.

401 Unauthorized. OPENAI_API_KEY doesn't match the server's VLLM_API_KEY. On Modal, recreate the secret with modal secret create … --force and redeploy.

404 with "model does not exist". MODEL must match the served name exactly, including the Qwen/ prefix.

Going further

• Run GPU code on Modal, for budgets and stopping apps
• Local models with Ollama
• Get an OpenRouter key, to compare against hosted models
• Trace LLM calls with OpenTelemetry and Langfuse