Edition of Saturday 21 November 2026

Building a Software Factory

Comments on “Building a Software Factory”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

Runway, a fintech software company, describes handing each stage of software delivery to agents: writing to green CI, review, risk-based approval, browser QA, fixing main, and coordination. It reports that agents drove 38% of October merges, median first-approval wait fell from over two hours to about five minutes, and AI-approved merges were reverted about one in a thousand against one in two hundred for human-approved ones. The figures are self-reported from internal data without external verification, and the mix of low-risk changes skews the revert comparison.

Read the original at runway.com. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 7 of 118 with a score of 0.72. The panel rated it highest on new information (0.89) and reader interest (0.87), lowest on technical depth (0.66). Placed No. 7, where the score alone would also have put it: close to “Linear changed code review for AI-written code”, No. 19 on Tuesday 17 November 2026 (−0.007). The placement terms move stories; they are not part of the score.

AI judgeJudge 2claude-haiku-5-5
Detailed, self-reported engineering account of agent-driven software delivery with concrete metrics, though mostly company-internal and unverifiable; some promotional framing and hiring pitch at the end.
AI judgeJudge 3gpt-5.6-luna
A detailed, metrics-backed engineering account of delegating software delivery stages to agents, including review, approval, QA, rollback, coordination, and operational guardrails. It is highly actionable, though evidence is internal and conclusions remain company-specific.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.72total under rubric v2.3

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.71
  • Reader interest0.87
  • Audience breadth0.71
  • Practical relevance0.81
  • Substantiated0.79
  • Technical depth0.66
  • New information0.89
  • Counts against: Promotional0.18
  • Counts against: Overclaim0.27

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.770.550.820.710.27+0.24+0.171
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.810.850.940.870.13+0.26+0.225
Audience breadthHow many people who build technology does it concern?0.550.800.780.710.25+0.13+0.092
Practical relevanceHow useful is it to someone who builds software or tech products?0.830.700.880.810.18+0.22+0.177
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.940.600.840.790.34+0.05+0.040
Technical depthHow technically substantive is it?0.760.450.780.660.33+0.05+0.033
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.840.900.930.890.09+0.05+0.045
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.110.350.080.180.27−0.19−0.034
OverclaimDo the headline or conclusions claim more than the body supports?0.240.400.180.270.22−0.11−0.030
Sum of contributions+0.719
Claim-check penalty0.000
Total, as published0.719

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #7 on Rubric, 21 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-21/7/"><img src="https://rubric.hrih.si/badge/2026-11-21/7.svg" alt="Ranked #7 on Rubric, 21 Nov 2026" width="238" height="28"></a>

Claim check

Claim check: 8 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. No source givenNumber

    Cloud agents drove 38% of the pull requests merged in October, up from 0.1% in June.

    The article gives no link or citation for it.

    The article says

    “Cloud agents drove 38% of the pull requests we merged in October, up from 0.1% in June”

    The source says

    No source to compare with.

  2. No source givenNumber

    Between September and mid-November, the median wait for a first approval fell from over two hours to about five minutes.

    The article gives no link or citation for it.

    The article says

    “the median wait for a first approval fell from over two hours to about five minutes”

    The source says

    No source to compare with.

  3. No source givenGeneralization

    Changes approved by an agent were reverted no more often than those approved by people.

    The article gives no link or citation for it.

    The article says

    “Changes approved by an agent were reverted no more often than those approved by people”

    The source says

    No source to compare with.

  4. No source givenNumber

    Across September and October, a person objected to only 15 of the 977 pull requests the AI reviewer approved, about 1.5%.

    The article gives no link or citation for it.

    The article says

    “a person objected to only 15 of the 977 pull requests it approved, about 1.5%”

    The source says

    No source to compare with.

  5. No source givenNumber

    Of pull requests merged on AI approval alone from October to November, one in a thousand was reverted, versus about one in two hundred for those people approved.

    The article gives no link or citation for it.

    The article says

    “only one in a thousand was reverted, compared with about one in two hundred for those people approved”

    The source says

    No source to compare with.

  6. No source givenNumber

    In its first four days, the QA loop found 86 confirmed bugs, and agents fixed all of them.

    The article gives no link or citation for it.

    The article says

    “In its first four days, the QA loop found 86 confirmed bugs, and agents fixed all of them”

    The source says

    No source to compare with.

  7. No source givenNumber

    In its first two weeks the main-watching agents saw about 140 alerts, all but one fixed without calling anyone, with a median of 23 minutes from red to green.

    The article gives no link or citation for it.

    The article says

    “In its first two weeks it saw about 140 alerts. All but one were fixed without the coordinator having to call anyone in, with a median of 23 minutes from…”

    The source says

    No source to compare with.

  8. No source givenNumber

    In its first two weeks after going live, the coordinator dispatched over 14,000 agent tasks, which led to about 1,900 pull requests.

    The article gives no link or citation for it.

    The article says

    “it dispatched over 14,000 agent tasks, from diagnosis and planning to review and fixes, which led to about 1,900 pull requests”

    The source says

    No source to compare with.

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 1 pts, not on front page

Discussion on Hacker News

Recorded for context only, checked 21 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “Building a Software Factory”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.