Speculative Self-Distillation Enables Efficient Knowledge Internalization
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Speculative Self-Distillation (SSD) is a method for training a language model to internalize a document it never saw in pretraining, by distilling from the same model given that document.