title: Distributed Inference and Fine-tuning of Large Language Models Over The
    Internet
summary: Large language models (LLMs) are useful in many NLP tasks and become more
  capable with size, with the best open-source models having over 50 billion
  parameters. However, using these 50B+ models requires high-end hardware, making
  them inaccessible to most researchers. In this work, we investigate methods for
  cost-efficient inference and fine-tuning of LLMs, comparing local and
  distributed strategies. We observe that a large enough model (50B+) can run
  efficiently even on geodistributed devices in a consumer-grade network. This
  could allow running LLM efficiently by pooling together idle compute resources
  of multiple research groups and volunteers. We address two open problems: (1)
  how to perform inference and fine-tuning reliably if any device can disconnect
  abruptly and (2) how to partition LLMs between devices with uneven hardware,
  joining and leaving at will. In order to do that, we develop special
  fault-tolerant inference algorithms and load-balancing protocols that
  automatically assign devices to maximize the total system throughput. We
  showcase these algorithms in Petals - a decentralized system that runs Llama 2
  (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for
  interactive generation. We evaluate the performance of our system in simulated
  conditions and a real-world setup spanning two continents.
paper_authors._id: 657a6c1f857f942849b2ee22
paper_authors.name: Alexander Borzunov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:34:29.506Z
paper_authors._id: 609147936885f3a4314a6aa2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620133776745-noauth.jpeg
paper_authors.fullname: Alexander Borzunov
paper_authors.type: user
paper_authors.user: borzunov
paper_authors._id: 657a6c1f857f942849b2ee23
paper_authors.name: Max Ryabinin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:34:36.948Z
paper_authors._id: 607d59fb921db717010c7ccc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625736058289-607d59fb921db717010c7ccc.png
paper_authors.fullname: Max Ryabinin
paper_authors.type: user
paper_authors.user: mryab
paper_authors._id: 657a6c1f857f942849b2ee24
paper_authors.name: Artem Chumachenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:34:43.404Z
paper_authors._id: 6329ce19cf7d40df9b3a8f0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663684111181-noauth.jpeg
paper_authors.fullname: Artem Chumachenko
paper_authors.type: user
paper_authors.user: artek0chumak
paper_authors._id: 657a6c1f857f942849b2ee25
paper_authors.name: Dmitry Baranchuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:34:51.131Z
paper_authors._id: 62b6cc49752323892323bc04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b6cc49752323892323bc04/gGBld1KJIP9AIpd81L3PC.jpeg
paper_authors.fullname: Dmitry Baranchuk
paper_authors.type: user
paper_authors.user: dbaranchuk
paper_authors._id: 657a6c1f857f942849b2ee26
paper_authors.name: Tim Dettmers
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:34:56.724Z
paper_authors._id: 61a8f52d89ea472e600a0a10
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1638462736111-noauth.png
paper_authors.fullname: Tim Dettmers
paper_authors.type: user
paper_authors.user: timdettmers
paper_authors._id: 657a6c1f857f942849b2ee27
paper_authors.name: Younes Belkada
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:35:04.275Z
paper_authors._id: 62441d1d9fdefb55a0b7d12c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
paper_authors.fullname: Younes Belkada
paper_authors.type: user
paper_authors.user: ybelkada
paper_authors._id: 657a6c1f857f942849b2ee28
paper_authors.name: Pavel Samygin
paper_authors._id: 657a6c1f857f942849b2ee29
paper_authors.name: Colin Raffel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:35:18.194Z
paper_authors._id: 6079c29765b9d0165cb18392
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1618592397610-noauth.jpeg
paper_authors.fullname: Colin Raffel
paper_authors.type: user
paper_authors.user: craffel
paper_title: Distributed Inference and Fine-tuning of Large Language Models Over The
    Internet
paper_summary: Large language models (LLMs) are useful in many NLP tasks and become more
  capable with size, with the best open-source models having over 50 billion
  parameters. However, using these 50B+ models requires high-end hardware, making
  them inaccessible to most researchers. In this work, we investigate methods for
  cost-efficient inference and fine-tuning of LLMs, comparing local and
  distributed strategies. We observe that a large enough model (50B+) can run
  efficiently even on geodistributed devices in a consumer-grade network. This
  could allow running LLM efficiently by pooling together idle compute resources
  of multiple research groups and volunteers. We address two open problems: (1)
  how to perform inference and fine-tuning reliably if any device can disconnect
  abruptly and (2) how to partition LLMs between devices with uneven hardware,
  joining and leaving at will. In order to do that, we develop special
  fault-tolerant inference algorithms and load-balancing protocols that
  automatically assign devices to maximize the total system throughput. We
  showcase these algorithms in Petals - a decentralized system that runs Llama 2
  (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for
  interactive generation. We evaluate the performance of our system in simulated
  conditions and a real-world setup spanning two continents.
paper_ai_keywords: LLMs NLP tasks fine-tuning control-efficient inference geodistributed devices fault-tolerant inference algorithms load-balancing protocols Petals Llama 2 BLOOM system throughput offloading interactive generation