Hardware oﬄoading. Hardware oﬄoading means transferring temporarily unneeded
data in LLM from faster accelerators to slower yet ample primary and secondary stor-
age, such as CPU memory and disk. These data are subsequently reloaded as needed.
This method allows large LLMs to operate eﬃciently on GPUs with restricted memory
capacity. However, the oﬄoading and reloading processes inherently introduce signif-
icant communication overhead. Therefore, eﬀective oﬄoading strategy plays a crucial
role in enhancing overall system eﬃciency. FlexGen [18] can achieve high throughput
by developing a linear programming-based search algorithm that can identify the opti-
mal oﬄoading strategy within a deﬁned search space of possible oﬄoading strategies.
FlexGen further improves throughput by compressing both the weights and KV cache
for LLMs down to 4 bits. Additionally, FlexGen can be extended to a multi-GPU
setting by adopting pipeline parallelism. DeepSpeed [17] introduced an innovative
Beyond Efficiency