Step-by-step answer
Core 60–90 min7 stepsBuild a block and verify parameter counts against real models
RMSNorm, SwiGLU, RoPE and a causal pre-norm block in PyTorch, checked for causality, plus a parameter counter that matches Hugging Face's GPT-2 exactly (124,439,808) and a tiny Hugging Face Llama exactly, and gives 8,030,261,248 for Llama 3 8B. The MLP's share of each block rises from 67% in GPT-2 to 81% in Llama 3.
Back to the exercise