Musings on long-context fine-tuning
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Hardik Meisheri describes fine-tuning Qwen3.6-27B on examples of up to 70K tokens, about 64K of which are a shared system prompt. After fixing memory, autotuning and FSDP hang problems, he packed 75 to 90 examples behind one computed prompt, which required restoring standalone position IDs and isolating the 48 recurrent layers. He reports a two-epoch run under 20 hours versus 5 to 7 days, though data mixes differed, and several figures lack sources.