The Benchmarking Crisis
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
AI-driven code generation is making database benchmarks like ClickBench easy to overfit, since tools can tune engines query by query, exploiting workload quirks such as hash-only string grouping, predicate caching, and clustered-data shortcuts that wouldn't matter in real use.