title: Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a
    Single GPU
summary: Recent advances in large language models have brought immense value to the
  world, with their superior capabilities stemming from the massive number of
  parameters they utilize. However, even the GPUs with the highest memory
  capacities, currently peaking at 80GB, are far from sufficient to accommodate
  these vast parameters and their associated optimizer states when conducting
  stochastic gradient descent-based optimization. One approach to hosting such
  huge models is to aggregate device memory from many GPUs. However, this
  approach introduces prohibitive costs for most academic researchers, who always
  have a limited budget for many high-end GPU servers. In this paper, we focus on
  huge model fine-tuning on a single, even low-end, GPU in a commodity server,
  which is accessible to most AI researchers. In such a scenario, the
  state-of-the-art work ZeRO-Infinity suffers from two severe issues when running
  in a commodity server: 1) low GPU utilization due to inefficient swapping, and
  2) limited trainable model size due to CPU memory capacity. The underlying
  reason is that ZeRO-Infinity is optimized for running on high-end GPU servers.
  To this end, we present Fuyou, a low-cost training framework that enables
  efficient 100B huge model fine-tuning on a low-end server with a low-end GPU
  and limited CPU memory capacity. The key idea is to add the SSD-CPU
  communication as an optimization dimension and thus carefully co-optimize
  computation and data swapping from a systematic approach to maximize GPU
  utilization. The experimental results show that 1) Fuyou is able to fine-tune
  175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while
  ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model,
  Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves
  45 TFLOPS.
paper_authors._id: 65efcd5497074b211656dce9
paper_authors.name: Changyue Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:10:32.644Z
paper_authors._id: 648fbcab2d4c99e8421905d5
paper_authors.avatarUrl: /avatars/69e0911b535db612513ca8e3961470b2.svg
paper_authors.fullname: changyueliao
paper_authors.type: user
paper_authors.user: crispig
paper_authors._id: 65efcd5497074b211656dcea
paper_authors.name: Mo Sun
paper_authors._id: 65efcd5497074b211656dceb
paper_authors.name: Zihan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T13:26:35.905Z
paper_authors._id: 658019d199d0f9358e85fe47
paper_authors.avatarUrl: /avatars/c2bcb49a6154a7beef15182e61d100f2.svg
paper_authors.fullname: Zihan Yang
paper_authors.type: user
paper_authors.user: yzhblind
paper_authors._id: 65efcd5497074b211656dcec
paper_authors.name: Kaiqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T13:26:33.380Z
paper_authors._id: 64e33b34618cd90997d7312a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/35gG0C_nWI6CCOtRY0Ugl.png
paper_authors.fullname: Cage Chiaki
paper_authors.type: user
paper_authors.user: chiakicage
paper_authors._id: 65efcd5497074b211656dced
paper_authors.name: Binhang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:11:58.552Z
paper_authors._id: 6307c8dda642348dc5e71c20
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
paper_authors.fullname: Binhang Yuan
paper_authors.type: user
paper_authors.user: biyuan
paper_authors._id: 65efcd5497074b211656dcee
paper_authors.name: Fei Wu
paper_authors._id: 65efcd5497074b211656dcef
paper_authors.name: Zeke Wang
paper_title: Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a
    Single GPU
paper_summary: Recent advances in large language models have brought immense value to the
  world, with their superior capabilities stemming from the massive number of
  parameters they utilize. However, even the GPUs with the highest memory
  capacities, currently peaking at 80GB, are far from sufficient to accommodate
  these vast parameters and their associated optimizer states when conducting
  stochastic gradient descent-based optimization. One approach to hosting such
  huge models is to aggregate device memory from many GPUs. However, this
  approach introduces prohibitive costs for most academic researchers, who always
  have a limited budget for many high-end GPU servers. In this paper, we focus on
  huge model fine-tuning on a single, even low-end, GPU in a commodity server,
  which is accessible to most AI researchers. In such a scenario, the
  state-of-the-art work ZeRO-Infinity suffers from two severe issues when running
  in a commodity server: 1) low GPU utilization due to inefficient swapping, and
  2) limited trainable model size due to CPU memory capacity. The underlying
  reason is that ZeRO-Infinity is optimized for running on high-end GPU servers.
  To this end, we present Fuyou, a low-cost training framework that enables
  efficient 100B huge model fine-tuning on a low-end server with a low-end GPU
  and limited CPU memory capacity. The key idea is to add the SSD-CPU
  communication as an optimization dimension and thus carefully co-optimize
  computation and data swapping from a systematic approach to maximize GPU
  utilization. The experimental results show that 1) Fuyou is able to fine-tune
  175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while
  ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model,
  Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves
  45 TFLOPS.
paper_ai_summary: Fuyou enables efficient fine-tuning of large models on low-end GPUs by optimizing SSD-CPU communication and computation, achieving higher GPU utilization compared to ZeRO-Infinity.
paper_ai_keywords: ZeRO-Infinity Fuyou GPU utilization SSD-CPU communication computation large models fine-tuning GPT-3 TFLOPS RTX 4090