You could have invented transformers from memory
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
This post argues that transformers' feedforward layers and self-attention are both associative memory stores, and that raising similarities to higher powers, or exponentiating them as softmax does, sharply increases how many memories can be recalled without interference.