Swapping LLMs in Under Two Seconds: A Sleep Mode for Llama.cpp
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A developer's llama-sleep wrapper for llama-swap parks idle llama.cpp models' GPU memory in host RAM instead of restarting them, cutting a 10.3-second model swap on a single RTX 3060 to 2.1 seconds while keeping cached prompts.