title: TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
summary: Large model inference is shifting from cloud to edge due to concerns about
  the privacy of user interaction data. However, edge devices often struggle with
  limited computing power, memory, and bandwidth, requiring collaboration across
  multiple devices to run and speed up LLM inference. Pipeline parallelism, the
  mainstream solution, is inefficient for single-user scenarios, while tensor
  parallelism struggles with frequent communications. In this paper, we argue
  that tensor parallelism can be more effective than pipeline on low-resource
  devices, and present a compute- and memory-efficient tensor parallel inference
  system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw
  data local in the users' devices and introduces a sliding window memory
  scheduler to dynamically manage layer weights during inference, with disk I/O
  latency overlapped with the computation and communication. This allows larger
  models to run smoothly on memory-limited devices. We analyze the communication
  bottleneck and find that link latency, not bandwidth, emerges as the main
  issue, so a star-based allreduce algorithm is implemented. Through extensive
  experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80%
  less time-to-first-token and token latency compared to Accelerate, and over 90%
  compared to Transformers and Galaxy, while cutting the peak memory footprint of
  Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.
paper_authors._id: 66fcb61ce97377a6e456c1f9
paper_authors.name: Zonghang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T09:13:26.241Z
paper_authors._id: 647466b8b68461d5cf795e3c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647466b8b68461d5cf795e3c/zaK6sdCbdPfYu14vg2Ty6.png
paper_authors.fullname: LIKirin
paper_authors.type: user
paper_authors.user: LIKirin
paper_authors._id: 66fcb61ce97377a6e456c1fa
paper_authors.name: Wenjiao Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T08:40:51.106Z
paper_authors._id: 6513c6c9c1fbded3b1977acc
paper_authors.avatarUrl: /avatars/d169b3462f952c5639e1471ed8bf84c9.svg
paper_authors.fullname: Wenjiao Feng
paper_authors.type: user
paper_authors.user: NeuronNomad
paper_authors._id: 66fcb61ce97377a6e456c1fb
paper_authors.name: Mohsen Guizani
paper_authors._id: 66fcb61ce97377a6e456c1fc
paper_authors.name: Hongfang Yu
paper_title: TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
paper_summary: Large model inference is shifting from cloud to edge due to concerns about
  the privacy of user interaction data. However, edge devices often struggle with
  limited computing power, memory, and bandwidth, requiring collaboration across
  multiple devices to run and speed up LLM inference. Pipeline parallelism, the
  mainstream solution, is inefficient for single-user scenarios, while tensor
  parallelism struggles with frequent communications. In this paper, we argue
  that tensor parallelism can be more effective than pipeline on low-resource
  devices, and present a compute- and memory-efficient tensor parallel inference
  system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw
  data local in the users' devices and introduces a sliding window memory
  scheduler to dynamically manage layer weights during inference, with disk I/O
  latency overlapped with the computation and communication. This allows larger
  models to run smoothly on memory-limited devices. We analyze the communication
  bottleneck and find that link latency, not bandwidth, emerges as the main
  issue, so a star-based allreduce algorithm is implemented. Through extensive
  experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80%
  less time-to-first-token and token latency compared to Accelerate, and over 90%
  compared to Transformers and Galaxy, while cutting the peak memory footprint of
  Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.
paper_ai_summary: A tensor parallel inference system named TPI-LLM improves large language model (LLM) inference on edge devices by minimizing memory usage and reducing latency.
paper_ai_keywords: large model inference edge devices pipeline parallelism tensor parallelism sliding window memory scheduler disk I/O latency star-based allreduce algorithm time-to-first-token token latency peak memory footprint