The story’s score and claims

Comments on No. 49 of 62 · Alex Wang · edition of Sunday 15 November 2026

You could have invented transformers from memory

Summary

This post argues that transformers' feedforward layers and self-attention are both associative memory stores, and that raising similarities to higher powers, or exponentiating them as softmax does, sharply increases how many memories can be recalled without interference.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules