The architecture of open-weight LLMs: MoE, hybrid layers, ternary weights

This study surveys the architectures behind today's open-weight language models, from dense Transformers to mixture-of-experts and hybrid designs that mix attention with recurrent or convolutional layers. Its central question is how each design trades memory, compute and quality, since these tradeoffs shape what runs on an 8 GB GPU.