Step-by-step answer
Core 60–90 min7 stepsImplement and stress-test a byte-level BPE tokenizer
A byte-level BPE tokenizer in plain Python with GPT-2's pre-split regex, trained on Pride and Prejudice at 300, 1,000 and 5,000 tokens and compared with tiktoken's o200k_base. Training updates pair counts incrementally, so a 5,000-token vocabulary trains in seconds, and a check script shows it learns the same merges as the slow textbook loop.
Back to the exercise