A founding document of mechanistic interpretability [Elhage et al. 2021]

Anthropic researchers, led by Nelson Elhage, published a mathematical framework for reverse-engineering transformers, restricted to toy models of two layers or fewer with no MLP blocks. Rewriting attention as independent, additive heads with separate QK and OV circuits lets bigram and skip-trigram tables be read from the weights. In two-layer models, composed heads form "induction heads" that account for in-context learning. The authors acknowledge that MLP layers remain poorly understood, and that the results are not yet shown for large models.