This weight does not spark joy
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Deletion tuning fine-tunes a language model by zeroing a very sparse set of weights in the feed-forward gate matrices, rather than updating values as full fine-tuning or LoRA do. Lu et al. report it forgets less and needs a much smaller mask.