The story’s score and claims

Comments on No. 4 of 46 · deepgains.substack.com · edition of Monday 9 November 2026

Model Surgery: Can an AI directly edit and make another AI forget facts?

Summary

Arun S, writing on Substack, tested whether Claude Opus 5.5 could make a fine-tuned 494M-parameter model forget 20 synthetic customers and 10 secret keys by editing weights directly, following an MIT CSAIL method. Claude changed 17 matrices; re-teaching restored 4% of forgotten facts, versus 3% for a model never trained on them, and other customers lost 2.5 points. Gradient ascent, NPO and refusal training damaged the model or only hid the facts. Results are the author's own, from one run, and Claude's mechanism explanation was partly wrong.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules