Larger Tokenizers Suffer More Multi-Epoch Regret
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A study on the Marin tokenizer finds that multi-epoch regret, the validation-loss gap between an 8-epoch run on one eighth of the text and a 1-epoch run on unique text, rises with vocabulary size.