Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts
  Models
Large Language Models (LLMs) based on Mixture-of-Experts (MoE) architecture
are showing promising performance on various tasks. However, running them on
resource-constrained settings, where GPU memory resources are not abundant, is
challenging due to huge model sizes. Existing systems that offload model
weights to CPU memory suffer from the significant overhead of frequently moving
data between CPU and GPU. In this paper, we propose Fiddler, a
resource-efficient inference engine with CPU-GPU orchestration for MoE models.
The key idea of Fiddler is to use the computation ability of the CPU to
minimize the data movement between the CPU and GPU. Our evaluation shows that
Fiddler can run the uncompressed Mixtral-8x7B model, which exceeds 90GB in
parameters, to generate over 3 tokens per second on a single GPU with 24GB
memory, showing an order of magnitude improvement over existing methods. The
code of Fiddler is publicly available at
https://github.com/efeslab/fiddler
65cad83520448f0d1230fa16
Keisuke Kamahori
claimed_verified
2024-02-13T15:33:35.770Z
6304ac1a412a1b9d381ca378
/avatars/f4724eb5afc2a3b0e61e6da7bfa7be27.svg
Keisuke Kamahori
user
kamahori
65cad83520448f0d1230fa17
Yile Gu
admin_assigned
2024-02-13T10:29:31.268Z
652bb9e7375b3a8bc17cbd75
https://cdn-avatars.huggingface.co/v1/production/uploads/652bb9e7375b3a8bc17cbd75/B98R_Lou9QJbDRBUfcvGP.jpeg
Michael Gu
user
yilegu
65cad83520448f0d1230fa18
Kan Zhu
65cad83520448f0d1230fa19
Baris Kasikci
admin_assigned
2024-02-13T10:29:51.940Z
654132fe5a9a913c6c870e79
/avatars/2f6807eddef1929c571977e9af35f952.svg
Baris Kasikci
user
kasikci
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts
  Models
Large Language Models (LLMs) based on Mixture-of-Experts (MoE) architecture
are showing promising performance on various tasks. However, running them on
resource-constrained settings, where GPU memory resources are not abundant, is
challenging due to huge model sizes. Existing systems that offload model
weights to CPU memory suffer from the significant overhead of frequently moving
data between CPU and GPU. In this paper, we propose Fiddler, a
resource-efficient inference engine with CPU-GPU orchestration for MoE models.
The key idea of Fiddler is to use the computation ability of the CPU to
minimize the data movement between the CPU and GPU. Our evaluation shows that
Fiddler can run the uncompressed Mixtral-8x7B model, which exceeds 90GB in
parameters, to generate over 3 tokens per second on a single GPU with 24GB
memory, showing an order of magnitude improvement over existing methods. The
code of Fiddler is publicly available at
https://github.com/efeslab/fiddler
Fiddler is an efficient inference engine for large MoE models that minimizes data movement between CPU and GPU, enabling high-speed inference on resource-constrained settings.
Mixture-of-Experts MoE resource-efficient inference CPU-GPU orchestration Mixtral-8x7B parameter-efficient fine-tuning