Faster CI with Orchestrated Pytest Workers
Comments on “Faster CI with Orchestrated Pytest Workers”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
Close, a company that sells a CRM, describes how it cut CI for its backend's roughly 58,000 mostly datastore-backed integration tests to about two and a half minutes, down from about 12 with static splitting across 17 runners. A central orchestrator now collects test files and hands out batches dynamically, so workers start backends in parallel and collect only what they run. Cheaper spot instances (54, plus 2 on-demand) cost about the same as before. These figures are Close's own and unverified externally.
Why it ranked
No. 19 of 144 with a score of 0.65. The panel rated it highest on technical depth (0.82) and practical relevance (0.82), lowest on significance (0.52). The judges disagreed most on overclaim (spread 0.49), so the story is marked contested. Placed No. 19, No. 38 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Practical engineering write-up on scaling pytest CI with an orchestrator and spot workers; concrete details and metrics, though results are self-reported and the topic is niche to test infrastructure.
- AI judgeJudge 3gpt-5.6-luna
- A detailed, evidence-backed engineering account of dynamically orchestrating pytest across many CI workers, including setup bottlenecks, Kubernetes coordination, failure reruns, balancing, spot instances, and observability. Highly actionable, though company-specific and overlapping broader CI-speed coverage.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.65total under rubric v2.3Contestedjudges differ by 0.41 on substantiated, 0.49 on overclaim
- Significance0.52
- Reader interest0.77
- Audience breadth0.62
- Practical relevance0.82
- Substantiated0.64
- Technical depth0.82
- New information0.77
- Counts against: Promotional0.10
- Counts against: Overclaim0.28
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.54 | 0.35 | 0.68 | 0.52 | 0.33 | +0.24 | +0.126 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.72 | 0.70 | 0.90 | 0.77 | 0.20 | +0.26 | +0.201 |
| Audience breadthHow many people who build technology does it concern? | 0.65 | 0.50 | 0.72 | 0.62 | 0.22 | +0.13 | +0.081 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.85 | 0.70 | 0.90 | 0.82 | 0.20 | +0.22 | +0.180 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.60 | 0.45 | 0.86 | 0.64 | 0.41 , above the contested threshold | +0.05 | +0.032 |
| Technical depthHow technically substantive is it? | 0.93 | 0.70 | 0.84 | 0.82 | 0.23 | +0.05 | +0.041 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.78 | 0.80 | 0.72 | 0.77 | 0.08 | +0.05 | +0.038 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.08 | 0.20 | 0.03 | 0.10 | 0.17 | −0.19 | −0.020 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.57 | 0.20 | 0.08 | 0.28 | 0.49 , above the contested threshold | −0.11 | −0.031 |
| Sum of contributions | +0.648 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.648 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-19/19/"><img src="https://rubric.hrih.si/badge/2026-11-19/19.svg" alt="Ranked #19 on Rubric, 19 Nov 2026" width="245" height="28"></a>Claim check
Claim check: 8 no source given. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
No source givenNumber
Close's primary backend service has nearly 58,000 automated tests.
The article gives no link or citation for it.
The article says “Quite a lot of them, nearly 58,000”
The source says No source to compare with.
No source givenNumber
Close's full suite of 58,000 integration tests runs in CI in about 2 and a half minutes.
The article gives no link or citation for it.
The article says “About 2 and a half minutes”
The source says No source to compare with.
No source givenDate
Close introduced tests in CI in 2012, using a Jenkins build script and the nose testing library.
The article gives no link or citation for it.
The article says “Tests in CI were introduced in 2012, with a Jenkins build script and the nose testing library”
The source says No source to compare with.
No source givenDate
Close introduced deterministic test splitting in late 2014, and that setup lasted about 12 years.
The article gives no link or citation for it.
The article says “we introduced it to Close in late 2014”
The source says No source to compare with.
No source givenNumber
Even at 17x parallelization, Close's CI times grew longer, and the best achievable wall clock runtime with the split setup was around 12 minutes.
The article gives no link or citation for it.
The article says “the best performance we could get out of this setup was around 12 minutes of wall clock runtime”
The source says No source to compare with.
No source givenVersion
currents.dev maintains a centralized queue of work for Playwright and changes the test runner to consume from that queue rather than from a deterministic split.
The article gives no link or citation for it.
The article says “they maintain a centralized queue of work and tweak the test runner logic to consume from that queue instead of from a deterministic -split of work”
The source says No source to compare with.
No source givenNumber
With spot instances, Close turned 17 on-demand runners into 2 on-demand runners plus 54 spot instances for about the same dollar cost.
The article gives no link or citation for it.
The article says “we were able to turn 17 on-demand runners into 2 on-demand runners + 54 spot instances for about the same dollar cost”
The source says No source to compare with.
No source givenCausal conclusion
Running multiple tests concurrently on the same database can cause tests to read each other's data, making multi-CPU parallelism unhelpful for Close.
The article gives no link or citation for it.
The article says “Running multiple tests at the same time on the same database can lead to logical conflicts where tests end up reading each other's data”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 2 pts, not on front page
Recorded for context only, checked 19 Nov 2026, 05:47 UTC. It played no part in the ranking.