Bf16 might be freezing your weight updates
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A blog post by mweiss17 argues that fine-tuning Mistral-7B with bf16 weights and a plain AdamW optimizer, as in the released Quiet-STaR code, effectively freezes most weights.