180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A Transformers pull request (huggingface/transformers#50717) adds an experts_offload option that streams a MoE model's routed experts from SSD through a bounded device cache, letting Qwen3.8-Flash-Next run on a 32 GB Mac.