Edition of Monday 16 November 2026

Sandboxes for AI Agents

Comments on “Sandboxes for AI Agents”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

Shawn, writing on Substack, lays out seven sandbox designs that let coding agents test their own changes and attach evidence to pull requests, from mocked single services to production-config preview environments and shadow traffic. Each trades realism against safety and speed. He favors ephemeral, per-task sandboxes, citing Stripe, Ramp and GitHub, though the claim that teams at scale have converged there is unsourced and the Copilot detail was unverified. He adds that test-account data, a dev-only login and recorded session evidence matter more than the design chosen.

Read the original at shmck.substack.com. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 8 of 60 with a score of 0.69. The panel rated it highest on practical relevance (0.88) and reader interest (0.76), lowest on substantiated (0.47).

AI judgeJudge 2claude-haiku-5-5
Practical, well-organized guide to sandboxing for coding agents with a taxonomy of designs and tradeoffs. Largely opinion and synthesis with some cited examples; little original data, so substantiation is moderate.}
AI judgeJudge 3gpt-5.6-luna
A detailed, actionable framework for safely validating agent-written code, covering isolation, data, authentication, multi-repo architecture, cost tradeoffs, and evidence. It is practice-informed but lacks reproducible measurements or a primary implementation.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.69total under rubric v2.3

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.58
  • Reader interest0.76
  • Audience breadth0.67
  • Practical relevance0.88
  • Substantiated0.47
  • Technical depth0.72
  • New information0.72
  • Counts against: Promotional0.06
  • Counts against: Overclaim0.13

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.670.350.720.580.37+0.24+0.139
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.790.600.900.760.30+0.26+0.198
Audience breadthHow many people who build technology does it concern?0.700.600.720.670.12+0.13+0.087
Practical relevanceHow useful is it to someone who builds software or tech products?0.940.800.900.880.14+0.22+0.194
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.340.400.680.470.34+0.05+0.024
Technical depthHow technically substantive is it?0.840.600.720.720.24+0.05+0.036
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.740.600.820.720.22+0.05+0.036
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.050.100.030.060.07−0.19−0.011
OverclaimDo the headline or conclusions claim more than the body supports?0.120.200.080.130.12−0.11−0.015
Sum of contributions+0.689
Claim-check penalty0.000
Total, as published0.689

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #8 on Rubric, 16 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-16/8/"><img src="https://rubric.hrih.si/badge/2026-11-16/8.svg" alt="Ranked #8 on Rubric, 16 Nov 2026" width="238" height="28"></a>

Claim check

Claim check: 5 matches source · 1 no source given · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. Matches sourceNumber

    Stripe's Minions run on pre-warmed devboxes that start in about 10 seconds.

    The cited source says the same thing.

    The article says

    “Stripe’s Minions run on pre-warmed devboxes that start in about 10 seconds”

    The source says

    “Devboxes are pre-warmed so one can be spun up in 10 seconds, with Stripe code and services pre-loaded”

    Source cited by the article: stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents

  2. Matches sourceNumber

    Ramp runs each session in a Modal sandbox booted from a snapshot rebuilt every 30 minutes.

    The cited source says the same thing.

    The article says

    “runs each session in a Modal sandbox booted from a snapshot rebuilt every 30 minutes”

    The source says

    “Every 30 minutes, a Modal Cron clones each of Ramp's repositories, installs all dependencies, runs initial builds, and saves a snapshot of the Sandbox's filesystem”

    Source cited by the article: modal.com/blog/how-ramp-built-a-full-context-background-coding-agent-on-modal

  3. Couldn’t checkVersion

    GitHub Copilot's coding agent works on an Actions runner behind a firewall allowlist.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “works on an Actions runner behind a firewall allowlist”

    The source says

    No passage could be compared.

    Source cited by the article: docs.github.com/copilot/concepts/coding-agent/about-copilot-coding-agent

  4. Matches sourceCausal conclusion

    Lyft ran full per-engineer environments until they stopped scaling, then moved to request routing.

    The cited source says the same thing.

    The article says

    “ran full per-engineer environments until they stopped scaling, then moved to request routing”

    The source says

    “Onebox eventually reached a point where it could not scale anymore”

    Source cited by the article: infoq.com/news/2022/03/scaling-testing-acceptance-lyft

  5. Matches sourceGeneralization

    Uber's SLATE and Lyft's staging overrides both run only the changed services and borrow the rest from a shared environment, routing tagged requests to the new version.

    The cited source says the same thing.

    The article says

    “Tagged requests reach your version; everything else hits the shared copies. Uber’s SLATE and Lyft’s staging overrides both work this way”

    The source says

    “SUT uses production instances of dependencies by default, for guaranteeing the best fidelity while ensuring production safety”

    Source cited by the article: uber.com/blog/simplifying-developer-testing-through-slate

  6. Matches sourceFirst, only or fastest

    Twitter open-sourced the shadow-traffic comparison approach as Diffy.

    The cited source says the same thing.

    The article says

    “Twitter open-sourced this as Diffy”

    The source says

    “Twitter has open sourced Diffy , an automated testing tool used in production for discovering potential bugs in new code running on Apache Trift and other HTTP-based services”

    Source cited by the article: infoq.com/news/2015/09/twitter-diffy

  7. Couldn’t checkVersion

    Ona and GitHub Codespaces both support cloning multiple repos into one sandbox.

    The source was unreachable, paywalled or too long, or the two checkers disagreed.

    The article says

    “Ona and GitHub Codespaces both support it”

    The source says

    No passage could be compared.

    Source cited by the article: ona.com/docs/ona/configuration/multi-repository

  8. No source givenGeneralization

    Teams running agents at scale have landed on ephemeral sandboxes.

    The article gives no link or citation for it.

    The article says

    “This is where teams running agents at scale have landed”

    The source says

    No source to compare with.

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 3 pts, not on front page

Discussion on Hacker News

Recorded for context only, checked 16 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “Sandboxes for AI Agents”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.

  1. Score 0.74

    Backdooring Codex