• Throughput quantiﬁes the model’s eﬃciency in processing requests. It deﬁnes
the rate at which LLMs can generate output tokens or complete tasks, typically
measured in tokens per second (TPS) [59, 188] or queries per second [205, 206].
A high throughput indicates an LLM’s ability to handle multiple requests quickly,
making it suitable for high-volume applications like real-time chatbots or large-scale
content generation.

• Speed-up ratio measures the improvement in inference speed compared to a base-
line model, typically expressed as a multiple (e.g., 1.2x, 3.5x) [54, 55]. The ratio
is calculated as the quotient of the baseline execution time to the time taken by
the improved LLM or the quotient of the improved LLM’s throughput to that of
the baseline model. The speed-up ratio draws a relative comparison between the
two models, which oﬀers a quantitative measure of progress in optimizing model
architectures, training algorithms, and hardware optimizations.

9.1.2 Memory
Beyond Efficiency