Move the Compute to the Data

In a Substack post, Carlos Morillo argues that Spark beat Hadoop MapReduce mainly through DAG scheduling and by no longer writing intermediate results to triple-replicated HDFS between steps, not through in-memory caching. His evidence is Spark's 2014 100 TB sort, which ran on disk, and figures from the RDD and NSDI 2015 papers, which the claim check could not verify. Caching still helped iterative workloads. For platform builders, the suggestion is to count how often a pipeline writes data it reads only once. The central thesis is his own interpretation.