title: Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts
    Models
summary: Large Language Models (LLMs) based on Mixture-of-Experts (MoE) architecture
  are showing promising performance on various tasks. However, running them on
  resource-constrained settings, where GPU memory resources are not abundant, is
  challenging due to huge model sizes. Existing systems that offload model
  weights to CPU memory suffer from the significant overhead of frequently moving
  data between CPU and GPU. In this paper, we propose Fiddler, a
  resource-efficient inference engine with CPU-GPU orchestration for MoE models.
  The key idea of Fiddler is to use the computation ability of the CPU to
  minimize the data movement between the CPU and GPU. Our evaluation shows that
  Fiddler can run the uncompressed Mixtral-8x7B model, which exceeds 90GB in
  parameters, to generate over 3 tokens per second on a single GPU with 24GB
  memory, showing an order of magnitude improvement over existing methods. The
  code of Fiddler is publicly available at
  https://github.com/efeslab/fiddler
paper_authors._id: 65cad83520448f0d1230fa16
paper_authors.name: Keisuke Kamahori
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T15:33:35.770Z
paper_authors._id: 6304ac1a412a1b9d381ca378
paper_authors.avatarUrl: /avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
paper_authors.fullname: Keisuke Kamahori
paper_authors.type: user
paper_authors.user: kamahori
paper_authors._id: 65cad83520448f0d1230fa17
paper_authors.name: Yile Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T10:29:31.268Z
paper_authors._id: 652bb9e7375b3a8bc17cbd75
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652bb9e7375b3a8bc17cbd75/B98R_Lou9QJbDRBUfcvGP.jpeg
paper_authors.fullname: Michael Gu
paper_authors.type: user
paper_authors.user: yilegu
paper_authors._id: 65cad83520448f0d1230fa18
paper_authors.name: Kan Zhu
paper_authors._id: 65cad83520448f0d1230fa19
paper_authors.name: Baris Kasikci
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T10:29:51.940Z
paper_authors._id: 654132fe5a9a913c6c870e79
paper_authors.avatarUrl: /avatars/2f6807eddef1929c571977e9af35f952.svg
paper_authors.fullname: Baris Kasikci
paper_authors.type: user
paper_authors.user: kasikci
paper_title: Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts
    Models
paper_summary: Large Language Models (LLMs) based on Mixture-of-Experts (MoE) architecture
  are showing promising performance on various tasks. However, running them on
  resource-constrained settings, where GPU memory resources are not abundant, is
  challenging due to huge model sizes. Existing systems that offload model
  weights to CPU memory suffer from the significant overhead of frequently moving
  data between CPU and GPU. In this paper, we propose Fiddler, a
  resource-efficient inference engine with CPU-GPU orchestration for MoE models.
  The key idea of Fiddler is to use the computation ability of the CPU to
  minimize the data movement between the CPU and GPU. Our evaluation shows that
  Fiddler can run the uncompressed Mixtral-8x7B model, which exceeds 90GB in
  parameters, to generate over 3 tokens per second on a single GPU with 24GB
  memory, showing an order of magnitude improvement over existing methods. The
  code of Fiddler is publicly available at
  https://github.com/efeslab/fiddler
paper_ai_summary: Fiddler is an efficient inference engine for large MoE models that minimizes data movement between CPU and GPU, enabling high-speed inference on resource-constrained settings.
paper_ai_keywords: Mixture-of-Experts MoE resource-efficient inference CPU-GPU orchestration Mixtral-8x7B parameter-efficient fine-tuning