The story’s score and claims

Comments on No. 10 of 60 · tensward.com · edition of Monday 16 November 2026

GPU says 35% busy. Busy doing what?

Summary

Tensward, which builds an open-source LLM inference tuning tool, found that nvidia-smi's 35% "busy" figure on an H100 running llama.cpp hid a 643 MB per-request copy of conversation state off the GPU for a hybrid Qwen model. Disabling prompt caching and checkpoints raised throughput from 58 to 110 tokens per second, but it slowed a returning-conversation workload on an 8B model by about 20%. The figures are the company's own and unsourced; its CUPTI-based counter is research tooling, and its crash rate was only measured on one H100.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules