Analysts followed AI agent's randomly assigned preferences in 86% of decisions
Comments on “Analysts followed AI agent's randomly assigned preferences in 86% of decisions”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Anthropic ran a randomized study of 50 professional analysts doing a farming-cooperative analysis with Claude Code, where each agent was secretly assigned a coin-flip lean on four open analytical choices. Analysts matched the lean 86% of the time, and 77% reused the agent's arguments but only 15% credited it. The effect on the final recommendation was smaller (54% vs 36%). The study covers one task, one model and deliberately strengthened leans, and the figures are the authors' own, not independently verified.
Why it ranked
No. 8 of 110 with a score of 0.68. The panel rated it highest on new information (0.92) and substantiated (0.84), lowest on technical depth (0.57). Placed No. 8, No. 14 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Rigorous randomized study of AI agent influence on expert decisions, with primary-source paper and clear design; highly relevant to AI tooling and research practice, though mostly conceptual for builders.
- AI judgeJudge 3gpt-5.6-luna
- Randomized study with professional analysts, quantified agent influence, methodological caveats, and implications for AI-assisted research and institutional oversight. It is highly relevant and well supported, though task and sample limitations constrain generalization.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.68total under rubric v2.3
- Significance0.63
- Reader interest0.79
- Audience breadth0.70
- Practical relevance0.66
- Substantiated0.84
- Technical depth0.57
- New information0.92
- Counts against: Promotional0.05
- Counts against: Overclaim0.19
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.72 | 0.40 | 0.78 | 0.63 | 0.38 | +0.24 | +0.152 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.71 | 0.70 | 0.95 | 0.79 | 0.25 | +0.26 | +0.205 |
| Audience breadthHow many people who build technology does it concern? | 0.81 | 0.50 | 0.78 | 0.70 | 0.31 | +0.13 | +0.091 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.64 | 0.55 | 0.80 | 0.66 | 0.25 | +0.22 | +0.146 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.94 | 0.70 | 0.88 | 0.84 | 0.24 | +0.05 | +0.042 |
| Technical depthHow technically substantive is it? | 0.74 | 0.35 | 0.62 | 0.57 | 0.39 | +0.05 | +0.028 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.88 | 0.90 | 0.98 | 0.92 | 0.10 | +0.05 | +0.046 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.05 | 0.10 | 0.01 | 0.05 | 0.09 | −0.19 | −0.010 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.26 | 0.20 | 0.12 | 0.19 | 0.14 | −0.11 | −0.021 |
| Sum of contributions | +0.678 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.678 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-10-29/8/"><img src="https://rubric.hrih.si/badge/2026-10-29/8.svg" alt="Ranked #8 on Rubric, 29 Oct 2026" width="238" height="28"></a>Claim check
Claim check: 8 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Couldn’t checkNumber
Across the four analytical choices, analysts went with their AI agent's randomly assigned lean 86% of the time.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Across the four analytical choices, analysts went with their agent’s lean 86% of the time”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkNumber
Among analysts who had already expressed a preference, they switched away from it 60% of the time when the agent disagreed, versus 6% when it agreed.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “they switched away from it 60% of the time if the agent disagreed, compared to 6% if it agreed”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkNumber
77% of analysts repeated the agent's arguments in their explanations, but only 15% credited the agent as the source.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “77% of analysts used the agents’ own arguments—sometimes word for word—but only 15% credited the agent as the source of those arguments”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkNumber
When the agent leaned toward rolling out the program, 54% of analysts recommended it, compared with 36% when it leaned the other way.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “When the agent leaned toward rolling out the program, 54% of analysts recommended it, compared with 36% when it leaned the other way”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkNumber
Six of 11 analysts who worked without the agent (55%) made serious methodological errors that invalidated their results, compared with three of 50 (6%) who used it.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “six (55%) made serious methodological errors that invalidated their results, compared with three of the 50 (6%) who did use it”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkNumber
Without an assigned lean, Claude Opus 4.8 made the same five choices in 112 out of 123 runs.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “with no lean assigned, Claude Opus 4.8 made the same five choices in 112 out of 123 runs”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkBenchmark result
A statistical model using only the four leans correctly predicted 82% of analysts' recommendations, versus 61% for always guessing the most common recommendation.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “a statistical model correctly predicted 82% of the analysts’ recommendations, compared with 61% for always guessing the most common recommendation”
The source says No passage could be compared.
Source cited by the article: www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf
Couldn’t checkGeneralization
When separate research teams all defer to models with the same preferences, their agreement may no longer be independent.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “If those teams all defer to models with the same preferences, their agreement may no longer be independent”
The source says No passage could be compared.
Source cited by the article: doi.org/10.1126/sciadv.adn5290
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 4 pts, not on front page
Recorded for context only, checked 29 Oct 2026, 05:47 UTC. It played no part in the ranking.