Evolving Infra: Agent-Adaptive LLM Serving at Scale
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Evolving Infra, an open-source LLM serving system from MIT Han Lab (arXiv 2611.12304), adapts to agent workloads online without offline profiling. It separates compute allocation from KV cache management, predicts prefill and decode load using a cost model covering sub-agents and context compaction, and evicts KV no agent branch needs.…