ShadowPrefill: Keeping Your Next Model Warm for Model Routing in Agentic Serving
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
ShadowPrefill is a proposed serving technique for agentic LLM workloads that addresses the cost of switching models mid-session in per-step routing. Because KV caches are model-specific, a switch forces the new model to re-prefill the session context, raising latency and reducing prefix-cache hits.