I ran 120B MoE models on my 32GB M1 Max using under 16 GB
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A developer ran 120-billion-parameter mixture-of-experts models on a 32GB M1 Max by keeping most expert weights on SSD under a 16 GiB process-memory budget. Their engine averaged 7.84 tokens/s versus 5.78 for TUFF 10.2.2 on matched BF16 weights, and degraded less when another process used memory.