Model Surgery: Can an AI directly edit and make another AI forget facts?
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Arun S, writing on Substack, tested whether Claude Opus 5.5 could make a fine-tuned 494M-parameter model forget 20 synthetic customers and 10 secret keys by editing weights directly, following an MIT CSAIL method. Claude changed 17 matrices; re-teaching restored 4% of forgotten facts, versus 3% for a model never trained on them, and other customers lost 2.5 points. Gradient ascent, NPO and refusal training damaged the model or only hid the facts. Results are the author's own, from one run, and Claude's mechanism explanation was partly wrong.