The story’s score and claims

Comments on No. 84 of 144 · GitHub · edition of Thursday 19 November 2026

Evolving Infra: Agent-Adaptive LLM Serving at Scale

Summary

Evolving Infra, an open-source LLM serving system from MIT Han Lab (arXiv 2611.12304), adapts to agent workloads online without offline profiling. It separates compute allocation from KV cache management, predicts prefill and decode load using a cost model covering sub-agents and context compaction, and evicts KV no agent branch needs.…

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules