Serving Open-Weight Models
Put the model you fine-tuned behind a production endpoint: inference engines, GPU sizing, quantisation, latency tuning, structured outputs, rollout and operations.
3 modules · 10 lessons~32 hoursUpdated 2026-11-13
- Lessons10
- Quizzes10
- Sources71
- Of study~32h
About this course
About this course
You arrive with a LoRA adapter on a small open-weight model and an eval suite, and leave with a dependency other services can call. The course explains what an inference engine does with your GPU during prefill and decode, how to size and quantise a model with evidence from your own evals, how to hit a latency target under load, how to get guaranteed JSON and tool calls from an open model, and how to deploy, roll out and run the endpoint with metrics, SLOs and a capacity plan. Every lesson has a GPU path (Modal, Kaggle or your own NVIDIA card) and a CPU path through llama.cpp, so you can finish without a GPU.
About the running example
The examples follow Quillmoor, a fictional project-management SaaS whose operations team fine-tuned a small open model to triage support tickets and draft reply outlines, and now has to serve it to the help desk, the help-centre chat and a nightly backlog job. It is the same company as in Fine-Tuning and RL for Agents, and the exercises serve an adapter for the base you choose (your own, or any LoRA adapter for that base from the Hub) or the base alone; the course ships its own tickets and fixtures and stands alone. Worked numbers use Qwen3-4B-Instruct-2507 on one L4 GPU. All tickets, accounts and numbers are synthetic.
What you'll learn.
- 1Explain what an inference engine does during prefill and decode, and predict from weights, bandwidth and batch size whether a step is memory- or compute-bound and what that does to time to first token and tokens per second
- 2Size a GPU for a model, context length and concurrency from weights times precision plus KV cache per token, and choose between one GPU, tensor parallel and pipeline parallel with a written memory budget
- 3Quantise a fine-tuned model to W4A16, FP8 or GGUF, and measure what it costs with your own eval suite and a KL-divergence check before shipping it
- 4Tune a vLLM or SGLang endpoint to a TTFT, inter-token latency and goodput target with concurrency sweeps, chunked prefill, prefix caching and speculative decoding, and benchmark it honestly
- 5Serve guaranteed JSON and tool calls from an open model with server-side guided decoding, and serve one or many LoRA adapters without merging
- 6Deploy the endpoint in a container with health checks, warm-up, autoscaling and a shadow-then-canary rollout, run it against SLOs with engine metrics, and write a capacity plan and a runbook
3 modules, 10 lessons.
14h 10m of lessons
Module 1 · 3 lessons · 4h 10m
What the server does with your GPU
How an inference engine turns weights and a KV cache into tokens, how much GPU that takes, and how your adapter gets in.
Module 2 · 4 lessons · 5h 45m
Smaller, faster, still correct
Quantisation you can defend with evals, latency tuning you can benchmark, speculative decoding, and guaranteed structure from the server.
Module 3 · 3 lessons · 4h 15m
Ship it and run it
What you take on when you self-host, how to deploy and roll out, and how to run the endpoint against SLOs.
Ship the triage endpoint
Take a fine-tuned triage model from an adapter on disk to a production endpoint with evidence at every step, from a quantised artefact proven by evals to a benchmarked, structured, canaried and monitored service with a capacity plan and a runbook.