The story’s score and claims

Comments on No. 16 of 60 · hardikmeisheri.github.io · edition of Monday 16 November 2026

Musings on long-context fine-tuning

Summary

Hardik Meisheri describes fine-tuning Qwen3.6-27B on examples of up to 70K tokens, about 64K of which are a shared system prompt. After fixing memory, autotuning and FSDP hang problems, he packed 75 to 90 examples behind one computed prompt, which required restoring standalone position IDs and isolating the 48 recurrent layers. He reports a two-epoch run under 20 hours versus 5 to 7 days, though data mixes differed, and several figures lack sources.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules