Your Experts Have Partners. There's HOPE for Keeping Them Together
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
HOPE is a second-order method for pruning experts from Mixture-of-Experts language models, described in a paper by its authors. Existing pruning methods score each expert independently, but HOPE accounts for pairwise interactions between co-activated experts, which matters most at aggressive pruning rates.