Module 1 · Inside the transformer · Lesson 1

Tokens and embeddings

How raw text is chopped into tokens with byte-pair encoding, and how each token becomes a vector the network can compute with.

45 min 2 videos 3 exercises 7-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv

By the end of this lesson you can

Run the byte-pair encoding algorithm by hand on a small corpus
Explain why models struggle with spelling, arithmetic and some non-English text
Describe the embedding matrix and what an embedding lookup actually computes
Use cosine similarity to reason about meaning as direction in vector space

A language model never sees text. It sees a sequence of integers, and it turns each integer into a list of a few thousand real numbers before anything interesting happens. This lesson covers both steps: tokenization (text to integers) and embedding (integers to vectors). They look like plumbing, but they quietly explain many behaviours that seem mysterious, from why models miscount the letters in "strawberry" to why some languages cost more per sentence than others.

Module 1 · Inside the transformer · Lesson 1

Tokens and embeddings

How raw text is chopped into tokens with byte-pair encoding, and how each token becomes a vector the network can compute with.

45 min 2 videos 3 exercises 7-question quiz Answers

Setup for the exercises:Install Claude Code Python with uv

By the end of this lesson you can

Run the byte-pair encoding algorithm by hand on a small corpus
Explain why models struggle with spelling, arithmetic and some non-English text
Describe the embedding matrix and what an embedding lookup actually computes
Use cosine similarity to reason about meaning as direction in vector space

Free · sign in to start

Sign in to keep reading.

Free, no card. We use your account to save progress, quiz scores, bookmarks and occasionally send updates from our team.

7 more sections
3 hands-on exercises
A 7-question quiz
Progress, quiz scores and bookmarks