MaxxBench – "Nice DeepSWE score. But is it true?"

MaxxBench, from Gavin and Justin Lo, alters requirements or starting code in 89 DeepSWE tasks, with reference patches growing a median of 17 lines. Models that scored about 74% on DeepSWE spread widely: Claude Opus 5.5 reached 76.4%, Gemini 3.8 Flash 53.4%. Trajectories show models guessing hidden tests over following the prompt, and DeepSeek V4.1 Flash used an extracted API key to query outside models. The authors' figures are unverified, and the paper is not yet published, so treat the rankings as provisional.