title: PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday
    Home Clusters
summary: Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers
  for running frontier large language models (LLMs) on home devices. While
  consumer hardware is getting stronger and model quantization is improving,
  existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high
  bandwidth, far beyond what a common home cluster can handle. This paper
  introduces prima.cpp, a distributed inference system that runs 70B-scale models
  on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and
  cross-platform support. It uses mmap to manage model weights and introduces
  piped-ring parallelism with prefetching to hide disk loading. By modeling
  heterogeneity in computation, communication, disk, memory (and its management
  behavior), and OS, it optimally assigns model layers to each device's CPU and
  GPU, further reducing token latency. An elegant algorithm named Halda is
  proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a
  common four-node home cluster. It outperforms llama.cpp, exo, and dllama on
  30B+ models while keeping memory pressure below 6%. This brings frontier
  30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home
  assistants, making advanced AI truly accessible to individuals. The code is
  open source and available at https://github.com/Lizonghang/prima.cpp.
paper_authors._id: 67fdbde764a418633ee9fa1b
paper_authors.name: Zonghang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:53.716Z
paper_authors._id: 647466b8b68461d5cf795e3c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647466b8b68461d5cf795e3c/zaK6sdCbdPfYu14vg2Ty6.png
paper_authors.fullname: LIKirin
paper_authors.type: user
paper_authors.user: LIKirin
paper_authors._id: 67fdbde764a418633ee9fa1c
paper_authors.name: Tao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:50.518Z
paper_authors._id: 669e0c108b279f0a2704a5ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/669e0c108b279f0a2704a5ba/uPyioyK0RWSG7CT52xBao.png
paper_authors.fullname: TaoLi
paper_authors.type: user
paper_authors.user: LiPhilip
paper_authors._id: 67fdbde764a418633ee9fa1d
paper_authors.name: Wenjiao Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:15:55.453Z
paper_authors._id: 6513c6c9c1fbded3b1977acc
paper_authors.avatarUrl: /avatars/d169b3462f952c5639e1471ed8bf84c9.svg
paper_authors.fullname: Wenjiao Feng
paper_authors.type: user
paper_authors.user: NeuronNomad
paper_authors._id: 67fdbde764a418633ee9fa1e
paper_authors.name: Mohsen Guizani
paper_authors._id: 67fdbde764a418633ee9fa1f
paper_authors.name: Hongfang Yu
paper_title: PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday
    Home Clusters
paper_summary: Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers
  for running frontier large language models (LLMs) on home devices. While
  consumer hardware is getting stronger and model quantization is improving,
  existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high
  bandwidth, far beyond what a common home cluster can handle. This paper
  introduces prima.cpp, a distributed inference system that runs 70B-scale models
  on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and
  cross-platform support. It uses mmap to manage model weights and introduces
  piped-ring parallelism with prefetching to hide disk loading. By modeling
  heterogeneity in computation, communication, disk, memory (and its management
  behavior), and OS, it optimally assigns model layers to each device's CPU and
  GPU, further reducing token latency. An elegant algorithm named Halda is
  proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a
  common four-node home cluster. It outperforms llama.cpp, exo, and dllama on
  30B+ models while keeping memory pressure below 6%. This brings frontier
  30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home
  assistants, making advanced AI truly accessible to individuals. The code is
  open source and available at https://github.com/Lizonghang/prima.cpp.
paper_ai_summary: Prima.cpp, a distributed inference system, enables running large language models on home devices using CPU/GPU and low memory, overcoming hardware limitations.
paper_ai_keywords: LLMs large language models mmap piped-ring parallelism prefetching token latency Halda home cluster memory pressure