Next-Level Benchmarking Overfitting

Thomas Neumann, an Umbra developer, argues that GenDB-V, an AI-generated database engine whose paper claims 22× to 61× speedups over Umbra and DuckDB, overfits to TPC-H. He says the gains come from per-query structures precomputed at load time, and that the "plain-array" mode hard-codes dbgen's row layout, with a slow fallback otherwise. His bandwidth estimate puts such speedups out of reach for conventional execution. Most figures are his own and unverified, so the takeaway is caution about benchmark-tuned agents.