Score Centering: Understanding and Fixing Instability in RL
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Cameron Wolfe's overview covers score centering, a method for stabilizing LLM reinforcement learning when the sampler and trainer disagree on token probabilities. Rather than importance sampling's multiplicative weights, it subtracts the sampler's expected score from the policy gradient, which the paper argues removes mismatch-induced drift. Qwen3-1.7B Countdown diagnostics show offline training is stable only with non-negative rewards, while online training is most stable with group-centered rewards. The headline gains, such as 30% versus 12% accuracy under severe quantization, have no independent source.