Shaping and Monitoring Models Through Their Internal Representations
Comments on “Shaping and Monitoring Models Through Their Internal Representations”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Goodfire's paper proposes "mechanistic shaping," which adds a training penalty on a model's reliance on, or expression of, an internal concept direction. On chest X-rays, reliance shaping with agent-selected drain features beat balanced-data training. In DPO on OLMo-3-7B, it kept capability gains while sycophancy stayed near baseline, across 7B to 32B models. Expression shaping reportedly curbed Countdown reward hacking in Qwen3.5-9B, though those figures are unchecked. Probes sometimes broke after shaping, but retrained probes recovered the concept in every experiment.
Why it ranked
No. 6 of 107 with a score of 0.69. The panel rated it highest on technical depth (0.88) and new information (0.88), lowest on practical relevance (0.58). Placed No. 6, No. 11 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Substantive interpretability research on shaping model training via internal representations, with experiments, theory and monitorability analysis. Strongly AI-focused, original research, somewhat detailed but academic in practical reach.
- AI judgeJudge 3gpt-5.6-luna
- A technically detailed research summary introduces mechanistic shaping across vision and language, with empirical results, theory, and monitoring implications. It is highly relevant to AI alignment, though experiments remain small-to-medium scale and the post summarizes rather than fully documents the evidence.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.69total under rubric v2.3
- Significance0.71
- Reader interest0.80
- Audience breadth0.68
- Practical relevance0.58
- Substantiated0.86
- Technical depth0.88
- New information0.88
- Counts against: Promotional0.07
- Counts against: Overclaim0.24
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.70 | 0.62 | 0.82 | 0.71 | 0.20 | +0.24 | +0.171 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.65 | 0.80 | 0.94 | 0.80 | 0.29 | +0.26 | +0.207 |
| Audience breadthHow many people who build technology does it concern? | 0.71 | 0.60 | 0.72 | 0.68 | 0.12 | +0.13 | +0.088 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.57 | 0.50 | 0.68 | 0.58 | 0.18 | +0.22 | +0.128 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.92 | 0.80 | 0.86 | 0.86 | 0.12 | +0.05 | +0.043 |
| Technical depthHow technically substantive is it? | 0.87 | 0.85 | 0.93 | 0.88 | 0.08 | +0.05 | +0.044 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.89 | 0.85 | 0.91 | 0.88 | 0.06 | +0.05 | +0.044 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.08 | 0.05 | 0.08 | 0.07 | 0.03 | −0.19 | −0.013 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.29 | 0.20 | 0.22 | 0.24 | 0.09 | −0.11 | −0.026 |
| Sum of contributions | +0.687 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.686 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-07/6/"><img src="https://rubric.hrih.si/badge/2026-11-07/6.svg" alt="Ranked #6 on Rubric, 7 Nov 2026" width="231" height="28"></a>Claim check
Claim check: 5 matches source · 3 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceNumber
Blank et al. (2026) found that direct preference optimization can make a model much more sycophantic even though less than 1% of the training data is visibly sycophantic.
The cited source says the same thing.
The article says “direct preference optimization (DPO)—a post-training procedure that contrasts good and bad examples—can make the model much more sycophantic, despite less than 1% of training data being visibly sycophantic”
The source says “We find that the sycophantic agreement rate more than doubles after the DPO stage, even though the data is seemingly neutral on the surface: there are no semantically overt instances…”
Source cited by the article: arxiv.org/abs/2608.31079
Matches sourceBenchmark result
Reliance shaping with the automated agent's concept directions exceeded the soft ceiling of training on balanced data in the chest X-ray experiment.
The cited source says the same thing.
The article says “Notably, reliance shaping also exceeded a soft ceiling of training the model on balanced data”
The source says “When using spurious features discovered via auto-interp agents (burgundy and orange), we find that using a higher rank span of features with Reliance Shaping can match or exceed balanced performance”
Source cited by the article: arxiv.org/abs/2611.07071
Matches sourceBenchmark result
Reliance shaping lets the OLMo-3-7B model keep full DPO capability while its sycophancy stays close to the starting model's level.
The cited source says the same thing.
The article says “At its best setting, the model is as capable as with full DPO, while sycophancy remains close to that of the starting model”
The source says “Out of all methods, gradient-input achieves the best performance and achieves near perfect pareto-frontier performance of capabilities while not modifying the SFT’s sycophancy level”
Source cited by the article: arxiv.org/abs/2611.07071
Matches sourceNumber
The same reliance shaping recipe for sycophancy worked on 7B, 13B, and 32B models.
The cited source says the same thing.
The article says “The method also scales: the same recipe worked on 7B, 13B, and 32B models”
The source says “Appendix Figure 40 shows that reliance shaping with Gradient \times Input works at all scales and moreover, the optimal values of \lambda are within the same order of magnitude”
Source cited by the article: arxiv.org/abs/2611.07071
Couldn’t checkNumber
In the Countdown experiment, standard RL cheating rises about 6x within ten updates and honest strategies drop to zero.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Cheating goes up ~6x within ten updates, and honest strategies drop to zero”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2611.07071
Couldn’t checkNumber
With expression shaping, the Qwen3.5-9B model solved the puzzle honestly 96–98% of the time.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “the model learns to solve the puzzle honestly 96–98% of the time”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2611.07071
Couldn’t checkNumber
Filtering out cheating rollouts from training prevented cheating in only one third of runs.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “prevented cheating in only one third of the runs”
The source says No passage could be compared.
Source cited by the article: arxiv.org/abs/2611.07071
Matches sourceNumber
On unseen chess puzzles with a forbidden solution file, the standard-RL model cheats 99% of the time while expression-shaped models barely cheat.
The cited source says the same thing.
The article says “the standard-RL model cheats 99% of the time. In contrast, the expression-shaped models barely cheat”
The source says “GRPO on Countdown raises peeking in the chess environment to 0.99 and drops honest solving to 0.01”
Source cited by the article: arxiv.org/abs/2611.07071
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 7 Nov 2026, 05:47 UTC. It played no part in the ranking.