Find an induction head in GPT-2

The score is a pure function of an attention pattern, so it lives in its own module and can be checked on hand-built patterns with no model at all. The scan script is then short: boot GPT-2, run the repeated sequences once with the attention patterns cached, apply the score to every head, plot. The decision that matters most is the score's definition: the query at position p should look at p − L + 1, the token after the earlier copy, not at p − L, the earlier copy itself.