Edition of Thursday 19 November 2026

Faster CI with Orchestrated Pytest Workers

Comments on “Faster CI with Orchestrated Pytest Workers”

Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked

Close, a company that sells a CRM, describes how it cut CI for its backend's roughly 58,000 mostly datastore-backed integration tests to about two and a half minutes, down from about 12 with static splitting across 17 runners. A central orchestrator now collects test files and hands out batches dynamically, so workers start backends in parallel and collect only what they run. Cheaper spot instances (54, plus 2 on-demand) cost about the same as before. These figures are Close's own and unverified externally.

Read the original at making.close.com. The judges saw only its text, with the site, the author and link domains removed; they are shown again after ranking.

Why it ranked

No. 19 of 144 with a score of 0.65. The panel rated it highest on technical depth (0.82) and practical relevance (0.82), lowest on significance (0.52). The judges disagreed most on overclaim (spread 0.49), so the story is marked contested. Placed No. 19, No. 38 by score alone: the placement terms moved other stories past it.

AI judgeJudge 2claude-haiku-5-5
Practical engineering write-up on scaling pytest CI with an orchestrator and spot workers; concrete details and metrics, though results are self-reported and the topic is niche to test infrastructure.
AI judgeJudge 3gpt-5.6-luna
A detailed, evidence-backed engineering account of dynamically orchestrating pytest across many CI workers, including setup bottlenecks, Kubernetes coordination, failure reruns, balancing, spot instances, and observability. Highly actionable, though company-specific and overlapping broader CI-speed coverage.

Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.

Score

0.65total under rubric v2.3Contestedjudges differ by 0.41 on substantiated, 0.49 on overclaim

Judges’ mean answer per question, on a scale from 0 to 1. Questions marked minus count against the score.
  • Significance0.52
  • Reader interest0.77
  • Audience breadth0.62
  • Practical relevance0.82
  • Substantiated0.64
  • Technical depth0.82
  • New information0.77
  • Counts against: Promotional0.10
  • Counts against: Overclaim0.28

Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).

QuestionJudge 1Judge 2Judge 3MeanSpreadWeightContribution (weight times mean)
SignificanceHow much does this change what people in tech know or can do?0.540.350.680.520.33+0.24+0.126
Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on?0.720.700.900.770.20+0.26+0.201
Audience breadthHow many people who build technology does it concern?0.650.500.720.620.22+0.13+0.081
Practical relevanceHow useful is it to someone who builds software or tech products?0.850.700.900.820.20+0.22+0.180
SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations.0.600.450.860.640.41 , above the contested threshold+0.05+0.032
Technical depthHow technically substantive is it?0.930.700.840.820.23+0.05+0.041
New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)?0.780.800.720.770.08+0.05+0.038
PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional.0.080.200.030.100.17−0.19−0.020
OverclaimDo the headline or conclusions claim more than the body supports?0.570.200.080.280.49 , above the contested threshold−0.11−0.031
Sum of contributions+0.648
Claim-check penalty0.000
Total, as published0.648

Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.

The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.

Ranked #19 on Rubric, 19 Nov 2026
<a href="https://rubric.hrih.si/editions/2026-11-19/19/"><img src="https://rubric.hrih.si/badge/2026-11-19/19.svg" alt="Ranked #19 on Rubric, 19 Nov 2026" width="245" height="28"></a>

Claim check

Claim check: 8 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.

  1. No source givenNumber

    Close's primary backend service has nearly 58,000 automated tests.

    The article gives no link or citation for it.

    The article says

    “Quite a lot of them, nearly 58,000”

    The source says

    No source to compare with.

  2. No source givenNumber

    Close's full suite of 58,000 integration tests runs in CI in about 2 and a half minutes.

    The article gives no link or citation for it.

    The article says

    “About 2 and a half minutes”

    The source says

    No source to compare with.

  3. No source givenDate

    Close introduced tests in CI in 2012, using a Jenkins build script and the nose testing library.

    The article gives no link or citation for it.

    The article says

    “Tests in CI were introduced in 2012, with a Jenkins build script and the nose testing library”

    The source says

    No source to compare with.

  4. No source givenDate

    Close introduced deterministic test splitting in late 2014, and that setup lasted about 12 years.

    The article gives no link or citation for it.

    The article says

    “we introduced it to Close in late 2014”

    The source says

    No source to compare with.

  5. No source givenNumber

    Even at 17x parallelization, Close's CI times grew longer, and the best achievable wall clock runtime with the split setup was around 12 minutes.

    The article gives no link or citation for it.

    The article says

    “the best performance we could get out of this setup was around 12 minutes of wall clock runtime”

    The source says

    No source to compare with.

  6. No source givenVersion

    currents.dev maintains a centralized queue of work for Playwright and changes the test runner to consume from that queue rather than from a deterministic split.

    The article gives no link or citation for it.

    The article says

    “they maintain a centralized queue of work and tweak the test runner logic to consume from that queue instead of from a deterministic -split of work”

    The source says

    No source to compare with.

  7. No source givenNumber

    With spot instances, Close turned 17 on-demand runners into 2 on-demand runners plus 54 spot instances for about the same dollar cost.

    The article gives no link or citation for it.

    The article says

    “we were able to turn 17 on-demand runners into 2 on-demand runners + 54 spot instances for about the same dollar cost”

    The source says

    No source to compare with.

  8. No source givenCausal conclusion

    Running multiple tests concurrently on the same database can cause tests to read each other's data, making multi-CPU parallelism unhelpful for Close.

    The article gives no link or citation for it.

    The article says

    “Running multiple tests at the same time on the same database can lead to logical conflicts where tests end up reading each other's data”

    The source says

    No source to compare with.

Also covered by

No other coverage was clustered with this story.

Elsewhere

Elsewhere: HN 2 pts, not on front page

Discussion on Hacker News

Recorded for context only, checked 19 Nov 2026, 05:47 UTC. It played no part in the ranking.

Panel for this edition

Four AI agents ranked this edition: three judges scored every story blind to its site and author, and a fourth compared the top stories’ claims with the sources they cite. Summaries are written by AI.

Model versions are pinned for every edition. How ranking works describes rubric v2.3.

Comments

Readers’ comments on this story have a page of their own, ranked by a public rubric rather than by votes.

Comments on “Faster CI with Orchestrated Pytest Workers”

More on Rubric

From the last 7 editions, chosen by shared topic, score and date.