Model Surgery: Can an AI directly edit and make another AI forget facts?
Comments on “Model Surgery: Can an AI directly edit and make another AI forget facts?”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Arun S, writing on Substack, tested whether Claude Opus 5.5 could make a fine-tuned 494M-parameter model forget 20 synthetic customers and 10 secret keys by editing weights directly, following an MIT CSAIL method. Claude changed 17 matrices; re-teaching restored 4% of forgotten facts, versus 3% for a model never trained on them, and other customers lost 2.5 points. Gradient ascent, NPO and refusal training damaged the model or only hid the facts. Results are the author's own, from one run, and Claude's mechanism explanation was partly wrong.
Why it ranked
No. 4 of 46 with a score of 0.68. The panel rated it highest on technical depth (0.91) and new information (0.91), lowest on significance (0.62). Placed No. 4, where the score alone would also have put it: close to “Hiding a Rule Inside Another Model's Weights”, No. 12 on Friday 6 November 2026 (−0.008). The placement terms move stories; they are not part of the score.
- AI judgeJudge 2claude-haiku-5-5
- Detailed, technically deep experiment on LLM unlearning via weight editing with clear controls and honest limitations; strong interest for AI builders, but narrow scope and single run limit breadth and significance.
- AI judgeJudge 3gpt-5.6-luna
- A technically detailed, independently evaluated model-unlearning experiment with reproducible setup, attacks, controls, ablations, and limitations. It is highly relevant to AI alignment and data erasure, though results are limited to one small model and one run.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.68total under rubric v2.3
- Significance0.62
- Reader interest0.83
- Audience breadth0.66
- Practical relevance0.65
- Substantiated0.82
- Technical depth0.91
- New information0.91
- Counts against: Promotional0.07
- Counts against: Overclaim0.26
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.62 | 0.45 | 0.78 | 0.62 | 0.33 | +0.24 | +0.148 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.75 | 0.80 | 0.93 | 0.83 | 0.18 | +0.26 | +0.215 |
| Audience breadthHow many people who build technology does it concern? | 0.70 | 0.55 | 0.72 | 0.66 | 0.17 | +0.13 | +0.085 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.66 | 0.50 | 0.78 | 0.65 | 0.28 | +0.22 | +0.142 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.91 | 0.72 | 0.84 | 0.82 | 0.19 | +0.05 | +0.041 |
| Technical depthHow technically substantive is it? | 0.98 | 0.85 | 0.90 | 0.91 | 0.13 | +0.05 | +0.046 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.85 | 0.90 | 0.97 | 0.91 | 0.12 | +0.05 | +0.045 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.06 | 0.12 | 0.02 | 0.07 | 0.10 | −0.19 | −0.013 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.26 | 0.35 | 0.18 | 0.26 | 0.17 | −0.11 | −0.029 |
| Sum of contributions | +0.680 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.680 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-09/4/"><img src="https://rubric.hrih.si/badge/2026-11-09/4.svg" alt="Ranked #4 on Rubric, 9 Nov 2026" width="231" height="28"></a>Claim check
Claim check: 2 matches source · 5 no source given · 1 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceBenchmark result
In the MIT CSAIL paper's Experiment 8, the editor GPT-6 Astra raised Qwen3.5-0.8B's MATH-500 accuracy from 7.9 percent to 48.1 percent by editing 0.016 percent of the weights.
The cited source says the same thing.
The article says “It fixed this by editing 0.016 percent of the weights. MATH-500 accuracy went from 7.9 percent to 48.1 percent”
The source says “With 50 forward passes, MATH-500 rose from 7.9% to 48.1% and GSM8K from 13.8% to 67.1%”
Source cited by the article: rikiyat.github.io/blog/weight-space-programming
Couldn’t checkNumber
The paper "Sparks of Weight-Space Programming" used only one editor model, OpenAI's GPT-6 Astra.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “The paper used only one editor, OpenAI’s GPT-6 Astra”
The source says No passage could be compared.
Source cited by the article: rikiyat.github.io/blog/weight-space-programming
Matches sourceNumber
The paper's editor is limited to no gradient descent, a fixed budget of 10, 25 or 50 forward passes, and must explain each step before taking it.
The cited source says the same thing.
The article says “It has a small, fixed budget of “forward passes”: 10, 25 or 50”
The source says “Astra gets: a Python kernel over the weights (no GPU), practice problems (7,469 from GSM8K and 7,275 from MATH, not the test sets), and 10, 25 or 50 forward passes”
Source cited by the article: rikiyat.github.io/blog/weight-space-programming
No source givenNumber
Claude Opus 5.5 changed 17 weight matrices in a 494-million-parameter model, after which it had forgotten 20 customers and 10 secret keys while other customers lost only 2.5 points of accuracy.
The article gives no link or citation for it.
The article says “Claude Opus 5.5 changed 17 weight matrices in a 494-million-parameter model”
The source says No source to compare with.
No source givenNumber
After a short re-teaching attack, 92 percent of forgotten facts returned in the original model, 3 percent in the answer key, and 4 percent after Claude's edit.
The article gives no link or citation for it.
The article says “After Claude’s edit, 4 percent come back”
The source says No source to compare with.
No source givenNumber
Refusal training did not erase the facts: the model still completed the sentence with the true city, and 97.5 percent of other facts returned after re-teaching five facts.
The article gives no link or citation for it.
The article says “Teach it five facts again, and 97.5 percent of the other facts come back”
The source says No source to compare with.
No source givenGeneralization
The three standard retraining methods (gradient ascent, NPO, refusal training) did not both forget the facts and preserve the model; they hid the facts or damaged the model.
The article gives no link or citation for it.
The article says “The three standard retraining methods did not do both. They hid the facts, or they damaged the model to remove them”
The source says No source to compare with.
No source givenCausal conclusion
The late-layer matrices do the work of Claude's edit, but Claude's explanation that the middle layers remove secrets and ticket IDs was wrong, as they remove nothing alone.
The article gives no link or citation for it.
The article says “Alone, they remove nothing. Not the secrets, and not the ticket IDs”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 9 Nov 2026, 05:46 UTC. It played no part in the ranking.