SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for SpecDec

SpecPilot, from researchers at Berkeley, Princeton, Stanford and NVIDIA, is an agentic loop that tunes speculative decoding for each deployment. A cost model locates the gap to a ceiling, parallel agents test fixes, and token forcing checks outputs against stock to catch lossy shortcuts. It found a lossless 12-bit weight format and tree verification for hybrid recurrent models. The authors report speedup over autoregressive decoding rising from 4.46x to 10.87x with one request, and 1.94x to 3.93x with batching. These figures are self-reported and have no independent source.