Key LLM Inference Metrics Explained in 5 Minutes
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
A five-minute primer defines nine LLM inference metrics, splitting each request into a time-to-first-token wait and a streaming phase. It gives formulas for per-request latency (TTFT, E2E, TPOT, ITL), percentile-based latency across requests, and server-level throughput (RPS, TPS), plus goodput, which counts only requests meeting latency targets.