Running a Language Model on a $35 FPGA

Nishta Hirani-Nelson's write-up describes running a 1.07M-parameter Transformer language model on a $35 iCE40UP5K FPGA with only open-source tooling and no softcore CPU. It is notable because the design fits in about 144 KB of memory and integer-only arithmetic, using a mixture-of-experts layer, rotary embeddings, and gated delta nets, generating…