Step-by-step answer
Stretch 45 min6 stepsProve RoPE is relative, numerically
RoPE in a dozen lines of numpy, with a check script showing that rotation preserves norms and that the q·k score is unchanged when both positions shift by the same amount, plus a plot of score against distance for base 10,000 and 500,000. A larger base slows every rotation, so a token's match with itself decays far more slowly over 4,096 positions.
Back to the exercise