setting by adopting pipeline parallelism. DeepSpeed [17] introduced an innovative
technique called ZeRO-Inference to minimize latency and enhance throughput. This
approach involves pining the model weights in either DRAM or NVMe and dynam-
ically streaming each layer into GPU memory for computation as required. For a
multi-GPUs environment, DeepSpeed leverages both tensor parallelism and pipeline
parallelism to attain optimal performance. FastServe [187] designs the oﬄoading strat-
egy based on the priority of the jobs. Key-value tensors associated with lower-priority
Beyond Efficiency