N-gram hinted language modeling: NanoGPT Speedrun in 6.7s
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A post on Hrih News describes a NanoGPT Speedrun record, cutting the time to train a GPT-2-level model on an 8xH100 node from 39.9 seconds to 6.7 seconds using retrieval-augmented n-gram hints.