The Role of Humans in Software Factories
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Software factories, where AI agents write, test and ship code with little human oversight, are drawing debate in 2026 over how much human review to keep. Evidence from Faros AI shows high-adoption teams with more bugs and much longer review times, and SlopCodeBench finds agent code degrades over iterations.
Score
0.62total under rubric v2.0
- Significance0.63
- Reader interest0.84
- Audience breadth0.83
- Practical relevance0.84
- Substantiated0.64
- Technical depth0.69
- New information0.84
- Counts against: Promotional0.05
- Counts against: Overclaim0.18
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| Significance | 0.67 | 0.55 | 0.68 | 0.63 | 0.13 | +0.25 | +0.159 |
| Reader interest | 0.76 | 0.85 | 0.90 | 0.84 | 0.14 | +0.20 | +0.167 |
| Audience breadth | 0.88 | 0.80 | 0.82 | 0.83 | 0.08 | +0.10 | +0.083 |
| Practical relevance | 0.92 | 0.75 | 0.84 | 0.84 | 0.17 | +0.15 | +0.126 |
| Substantiated | 0.60 | 0.60 | 0.72 | 0.64 | 0.12 | +0.10 | +0.064 |
| Technical depth | 0.78 | 0.60 | 0.70 | 0.69 | 0.18 | +0.10 | +0.069 |
| New information | 0.93 | 0.60 | 0.98 | 0.84 | 0.38 | +0.10 | +0.084 |
| Promotional | 0.07 | 0.05 | 0.04 | 0.05 | 0.03 | −0.15 | −0.008 |
| Overclaim | 0.15 | 0.20 | 0.18 | 0.18 | 0.05 | −0.15 | −0.027 |
| Sum of contributions | +0.717 | ||||||
| Claim-check penalty | −0.100 | ||||||
| Total, as published | 0.617 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
Claim check
Claim check: 1 matches source · 1 differs from source · 3 no source given · 3 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Differs from sourceNumber
Faros AI found that high-AI-adoption teams had 54% more bugs per developer, incidents per PR up by more than 240%, and median review time roughly five times longer, while completing about a third more tasks.
The cited source says something else; both quotes are shown.
The article says “The high-adoption teams completed about a third more tasks, but they also had 54% more bugs per developer, incidents per PR were up by more than 240%, and median review…”
The source says “28% Bugs per PR”
Source cited by the article: faros.ai/research/ai-acceleration-whiplash
Couldn’t checkNumber
Faros AI analysed two years of data from 22,000 developers across 4,000 teams, comparing high-AI-adoption and low-adoption teams.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Faros AI looked at two years of data from 22,000 developers across 4,000 teams and compared teams with high AI adoption to teams with low adoption”
The source says No passage could be compared.
Source cited by the article: faros.ai/research/ai-acceleration-whiplash
No source givenNumber
A July 2026 IDC survey of 309 respondents found only 28% of organizations using AI coding assistants require peer review of AI-assisted changes.
The article gives no link or citation for it.
The article says “A July 2026 IDC survey of 309 respondents found that only 28% of organizations using AI coding assistants require peer review of AI-assisted changes”
The source says No source to compare with.
Couldn’t checkCausal conclusion
HumanLayer ran an experiment in July 2025 in which nobody read the AI-generated code, resulting in an outage, angry users, and a codebase humans hadn't looked at in three months.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “ran an experiment in July 2025 where nobody in their team read the AI-generated code”
The source says No passage could be compared.
Source cited by the article: youtube.com/watch
No source givenBenchmark result
SlopCodeBench results: across 36 problems and 196 checkpoints, structural erosion rose in 77% of runs and verbosity rose in about 75%.
The article gives no link or citation for it.
The article says “Across 36 problems and 196 checkpoints, structural erosion rose in 77% of runs and verbosity rose in about 75%”
The source says No source to compare with.
No source givenBenchmark result
On SlopCodeBench, agent code was about 2.3 times more verbose and twice as eroded as 473 open-source Python repositories, and the best strict solve rate was 14.8%.
The article gives no link or citation for it.
The article says “Compared against 473 open-source Python repositories, agent code was about 2.3 times more verbose and twice as eroded, and the best strict solve rate was 14.8%”
The source says No source to compare with.
Couldn’t checkNumber
Lauren Tan says she ships more than 2000 PRs a month.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “She talked about how she is able to ship >2000 PRs a month”
The source says No passage could be compared.
Source cited by the article: x.com/poteto/status/2102050467505430555
Matches sourceQuote
StrongDM's stated rules say humans must not write or review code, and the company claims to have practised this for product development.
The cited source says the same thing.
The article says “they have a rule on their factory pagesaying no humans should write or review code”
The source says “Code must not be written by humans - Code must not be reviewed by humans”
Source cited by the article: factory.strongdm.ai
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 24 Oct 2026, 08:54 UTC. It played no part in the ranking.