the softmax function only after all attention values have been determined. Building upon FlashAttention [61], FlashAttention-
2 [59] further optimizes work partitioning and reduces non-matrix multiplications, capitalizing on the GPUs’ optimization for
matrix operations. These algorithms are tailored to hardware considerations and accelerate the models on GPU machines.
In the quest to enhance Large Language Model (LLM) systems, some researchers are creatively drawing inspiration from
current hardware architectures. A notable example is the PagedAttention [144], which adapts virtual memory and paging
techniques, commonly used in operating systems, to overcome memory limitations in LLMs. PagedAttention introduces an
innovative approach to memory management by emulating the virtual memory system. It segments the Key-Value (KV) cache
associated with a request into blocks, as opposed to relying on pre-allocated, contiguous memory. This method significantly
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey