Module 1 · Inside the transformer · Lesson 3
The transformer block
The parts of the block that are not attention, namely the MLP, residual connections, normalisation and rotary position embeddings, and how they add up to a model whose parameters you can count from its config.
By the end of this lesson you can
- Write the equations of a modern pre-norm block (RMSNorm, attention, SwiGLU MLP, residual adds) and explain each design choice
- Explain the MLP as a key-value memory and why most of a dense model's parameters live there
- Show with a worked example why rotary position embeddings make attention scores depend only on relative position
- Count the parameters of GPT-2 small and Llama 3 8B from their hyperparameters
- Distinguish total from active parameters in a mixture-of-experts model