Move the Compute to the Data
Comments on “Move the Compute to the Data”
Claim-checked. Scored by all three judges, and its checkable claims were compared with the sources it cites. How far stories are checked
In a Substack post, Carlos Morillo argues that Spark beat Hadoop MapReduce mainly through DAG scheduling and by no longer writing intermediate results to triple-replicated HDFS between steps, not through in-memory caching. His evidence is Spark's 2014 100 TB sort, which ran on disk, and figures from the RDD and NSDI 2015 papers, which the claim check could not verify. Caching still helped iterative workloads. For platform builders, the suggestion is to count how often a pipeline writes data it reads only once. The central thesis is his own interpretation.
Why it ranked
No. 14 of 137 with a score of 0.67. The panel rated it highest on new information (0.92) and technical depth (0.90), lowest on significance (0.56). Placed No. 14, No. 20 by score alone: the placement terms moved other stories past it.
- AI judgeJudge 2claude-haiku-5-5
- Substantive, well-sourced essay on Spark vs MapReduce data movement with primary citations and technical depth; opinion-led series part, relevant to data engineers, not news-breaking, some framing claims are arguable.
- AI judgeJudge 3gpt-5.6-luna
- A deeply sourced technical analysis of MapReduce, Spark, data movement, DAGs, lineage, shuffles, and caching, with actionable platform guidance. Its strong thesis slightly overstates how singularly materialization explains Spark’s gains.
Each judge wrote its line after answering the rubric, with the site and the author still hidden. Every answer behind this is in the table under Score.
Score
0.67total under rubric v2.3
- Significance0.56
- Reader interest0.77
- Audience breadth0.67
- Practical relevance0.70
- Substantiated0.86
- Technical depth0.90
- New information0.92
- Counts against: Promotional0.04
- Counts against: Overclaim0.29
Each judge’s answer per question (0–1), their mean and spread, and what the question adds to the score (weight × mean).
| Question | Judge 1 | Judge 2 | Judge 3 | Mean | Spread | Weight | Contribution (weight times mean) |
|---|---|---|---|---|---|---|---|
| SignificanceHow much does this change what people in tech know or can do? | 0.64 | 0.35 | 0.68 | 0.56 | 0.33 | +0.24 | +0.134 |
| Reader interestWould a well-informed senior engineer or tech founder want to read this today, and likely pass it on? | 0.74 | 0.70 | 0.88 | 0.77 | 0.18 | +0.26 | +0.201 |
| Audience breadthHow many people who build technology does it concern? | 0.74 | 0.60 | 0.67 | 0.67 | 0.14 | +0.13 | +0.087 |
| Practical relevanceHow useful is it to someone who builds software or tech products? | 0.74 | 0.55 | 0.82 | 0.70 | 0.27 | +0.22 | +0.155 |
| SubstantiatedHow well is it backed? Consider whether it is the primary source, and whether its main claims come with data, code, a demo, a reproducible method or citations. | 0.83 | 0.85 | 0.91 | 0.86 | 0.08 | +0.05 | +0.043 |
| Technical depthHow technically substantive is it? | 0.96 | 0.80 | 0.94 | 0.90 | 0.16 | +0.05 | +0.045 |
| New informationDoes it contain information absent from the stories covered in the last 7 days (list provided)? | 0.91 | 0.90 | 0.96 | 0.92 | 0.06 | +0.05 | +0.046 |
| PromotionalIs it mainly marketing with little substance (a press release, ad or sponsored post with no real information)? A launch post that carries real news is not promotional. | 0.06 | 0.05 | 0.02 | 0.04 | 0.04 | −0.19 | −0.008 |
| OverclaimDo the headline or conclusions claim more than the body supports? | 0.24 | 0.30 | 0.32 | 0.29 | 0.08 | −0.11 | −0.032 |
| Sum of contributions | +0.671 | ||||||
| Claim-check penalty | 0.000 | ||||||
| Total, as published | 0.671 | ||||||
Spread is the highest minus the lowest judge answer. marks a spread above 0.40: the judges disagree, and the story is marked contested.
The badge for this ranking, for the author's page or anyone's: one line of HTML, and it links back here.
<a href="https://rubric.hrih.si/editions/2026-11-04/14/"><img src="https://rubric.hrih.si/badge/2026-11-04/14.svg" alt="Ranked #14 on Rubric, 4 Nov 2026" width="238" height="28"></a>Claim check
Claim check: 2 matches source · 4 no source given · 2 couldn’t check. Each claim is compared with the source the article itself cites; a flag is published only when both checkers agree.
Matches sourceBenchmark result
In 2014 Spark sorted 100 terabytes in 23 minutes on 206 machines, versus the previous Hadoop MapReduce record of 72 minutes on 2,100 machines.
The cited source says the same thing.
The article says “In 2014, Spark sorted 100 terabytes in 23 minutes on 206 machines. The previous record, held by Hadoop MapReduce, was 72 minutes on 2,100 machines”
The source says “using Spark on 206 EC2 machines, we sorted 100 TB of data on disk in 23 minutes. In comparison, the previous world record set by Hadoop MapReduce used 2100 machines…”
Source cited by the article: databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html
Matches sourceQuote
Databricks' announcement says the Spark sort record was set without using Spark's in-memory cache.
The cited source says the same thing.
The article says “Databricks’ own announcement says the record was set “without using Spark’s in-memory cache”
The source says “All the sorting took place on disk ( HDFS ), without using Spark’s in-memory cache”
Source cited by the article: databricks.com/blog/2014/11/05/spark-officially-sets-a-new-record-in-large-scale-sorting.html
Couldn’t checkNumber
In Jeff Dean's 2009 numbers, local disk delivers 200 MB/s while memory on a machine in another rack delivers 10 MB/s.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Local disk delivers 200 MB/s. Memory on a machine in another rack delivers 10 MB/s”
The source says No passage could be compared.
Source cited by the article: cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
Couldn’t checkQuote
The HDFS design document states the principle 'Moving Computation is Cheaper than Moving Data', and the default replication factor is three.
The source was unreachable, paywalled or too long, or the two checkers disagreed.
The article says “Moving Computation is Cheaper than Moving Data,” and the default replication factor is three”
The source says No passage could be compared.
Source cited by the article: hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
No source givenBenchmark result
In the RDD paper's logistic regression test on 100 machines, each Spark iteration was 25.3 times faster than Hadoop and 20.7 times faster than Hadoop with data in RAM.
The article gives no link or citation for it.
The article says “each Spark iteration was 25.3 times faster than Hadoop and 20.7 times faster than Hadoop with its data in RAM”
The source says No source to compare with.
No source givenNumber
Ousterhout et al. (NSDI 2015) found improving the network could improve job completion time by a median of at most 2%, and eliminating disk access would cut job time by a median of at most 19%.
The article gives no link or citation for it.
The article says “improving the network could improve job completion time “by a median of at most 2%.” Eliminating disk access entirely would have cut job time by a median of at most…”
The source says No source to compare with.
No source givenBenchmark result
In the RDD paper's experiment, logistic regression over 100 GB on 25 machines took 68.8 seconds per iteration with no data cached and 11.5 seconds with all cached.
The article gives no link or citation for it.
The article says “logistic regression over 100 GB on 25 machines took 68.8 seconds per iteration with none of the data cached and 11.5 seconds with all of it cached”
The source says No source to compare with.
No source givenCausal conclusion
Spark's speed over MapReduce came from DAG scheduling and ending materialization to replicated storage between steps, not from in-memory caching.
The article gives no link or citation for it.
The article says “Spark’s in-memory story got the credit. The real win was DAG scheduling and the end of materialization to replicated storage between steps”
The source says No source to compare with.
Also covered by
No other coverage was clustered with this story.
Elsewhere
Elsewhere: HN 1 pts, not on front page
Recorded for context only, checked 4 Nov 2026, 05:47 UTC. It played no part in the ranking.