Module 2 · How models are trained · Lesson 4

Pretraining and scaling laws

How one objective, predicting the next token over trillions of tokens of text, produces a base model, and how scaling laws decide model size, data and compute.

70 min 2 videos 3 exercises 6-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv Run Hugging Face models on CPU

By the end of this lesson you can

Compute the cross-entropy loss for a next-token prediction and explain why one sequence yields many training examples
Describe the main stages of a pretraining data pipeline and how they shape what a model knows
Estimate training compute with C ≈ 6ND and convert it into GPU-hours
Explain the Kaplan and Chinchilla scaling results and why modern small models are deliberately "over-trained"
Predict how a base model responds to a prompt, and why it is not yet an assistant

Everything an LLM can do starts from one objective: given some text, predict the next token. Train a transformer on that objective over a large fraction of the public internet and you get a base model, a system that has absorbed grammar, facts, code, legal boilerplate and styles of reasoning, but that does not yet answer questions or follow instructions. This lesson covers that first and most expensive stage: the objective, the data, the compute, and the scaling laws that decide how big a model should be.

Module 2 · How models are trained · Lesson 4

Pretraining and scaling laws

How one objective, predicting the next token over trillions of tokens of text, produces a base model, and how scaling laws decide model size, data and compute.

70 min 2 videos 3 exercises 6-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv Run Hugging Face models on CPU

By the end of this lesson you can

Compute the cross-entropy loss for a next-token prediction and explain why one sequence yields many training examples
Describe the main stages of a pretraining data pipeline and how they shape what a model knows
Estimate training compute with C ≈ 6ND and convert it into GPU-hours
Explain the Kaplan and Chinchilla scaling results and why modern small models are deliberately "over-trained"
Predict how a base model responds to a prompt, and why it is not yet an assistant

Free · sign in to start

Sign in to keep reading.

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

7 more sections
3 hands-on exercises
A 6-question quiz
Progress, quiz scores and bookmarks