Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a
  Single GPU
Recent advances in large language models have brought immense value to the
world, with their superior capabilities stemming from the massive number of
parameters they utilize. However, even the GPUs with the highest memory
capacities, currently peaking at 80GB, are far from sufficient to accommodate
these vast parameters and their associated optimizer states when conducting
stochastic gradient descent-based optimization. One approach to hosting such
huge models is to aggregate device memory from many GPUs. However, this
approach introduces prohibitive costs for most academic researchers, who always
have a limited budget for many high-end GPU servers. In this paper, we focus on
huge model fine-tuning on a single, even low-end, GPU in a commodity server,
which is accessible to most AI researchers. In such a scenario, the
state-of-the-art work ZeRO-Infinity suffers from two severe issues when running
in a commodity server: 1) low GPU utilization due to inefficient swapping, and
2) limited trainable model size due to CPU memory capacity. The underlying
reason is that ZeRO-Infinity is optimized for running on high-end GPU servers.
To this end, we present Fuyou, a low-cost training framework that enables
efficient 100B huge model fine-tuning on a low-end server with a low-end GPU
and limited CPU memory capacity. The key idea is to add the SSD-CPU
communication as an optimization dimension and thus carefully co-optimize
computation and data swapping from a systematic approach to maximize GPU
utilization. The experimental results show that 1) Fuyou is able to fine-tune
175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while
ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model,
Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves
45 TFLOPS.
65efcd5497074b211656dce9
Changyue Liao
admin_assigned
2024-03-12T12:10:32.644Z
648fbcab2d4c99e8421905d5
/avatars/69e0911b535db612513ca8e3961470b2.svg
changyueliao
user
crispig
65efcd5497074b211656dcea
Mo Sun
65efcd5497074b211656dceb
Zihan Yang
claimed_verified
2024-03-13T13:26:35.905Z
658019d199d0f9358e85fe47
/avatars/c2bcb49a6154a7beef15182e61d100f2.svg
Zihan Yang
user
yzhblind
65efcd5497074b211656dcec
Kaiqi Chen
claimed_verified
2024-03-13T13:26:33.380Z
64e33b34618cd90997d7312a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/35gG0C_nWI6CCOtRY0Ugl.png
Cage Chiaki
user
chiakicage
65efcd5497074b211656dced
Binhang Yuan
admin_assigned
2024-03-12T12:11:58.552Z
6307c8dda642348dc5e71c20
https://cdn-avatars.huggingface.co/v1/production/uploads/1661454675064-6307c8dda642348dc5e71c20.jpeg
Binhang Yuan
user
biyuan
65efcd5497074b211656dcee
Fei Wu
65efcd5497074b211656dcef
Zeke Wang
Adding NVMe SSDs to Enable and Accelerate 100B Model Fine-tuning on a
  Single GPU
Recent advances in large language models have brought immense value to the
world, with their superior capabilities stemming from the massive number of
parameters they utilize. However, even the GPUs with the highest memory
capacities, currently peaking at 80GB, are far from sufficient to accommodate
these vast parameters and their associated optimizer states when conducting
stochastic gradient descent-based optimization. One approach to hosting such
huge models is to aggregate device memory from many GPUs. However, this
approach introduces prohibitive costs for most academic researchers, who always
have a limited budget for many high-end GPU servers. In this paper, we focus on
huge model fine-tuning on a single, even low-end, GPU in a commodity server,
which is accessible to most AI researchers. In such a scenario, the
state-of-the-art work ZeRO-Infinity suffers from two severe issues when running
in a commodity server: 1) low GPU utilization due to inefficient swapping, and
2) limited trainable model size due to CPU memory capacity. The underlying
reason is that ZeRO-Infinity is optimized for running on high-end GPU servers.
To this end, we present Fuyou, a low-cost training framework that enables
efficient 100B huge model fine-tuning on a low-end server with a low-end GPU
and limited CPU memory capacity. The key idea is to add the SSD-CPU
communication as an optimization dimension and thus carefully co-optimize
computation and data swapping from a systematic approach to maximize GPU
utilization. The experimental results show that 1) Fuyou is able to fine-tune
175B GPT-3 on a consumer GPU RTX 4090 with high GPU utilization, while
ZeRO-Infinity fails to fine-tune; and 2) when training a small GPT-3 13B model,
Fuyou achieves 156 TFLOPS on an RTX 4090 GPU while ZeRO-Infinity only achieves
45 TFLOPS.
Fuyou enables efficient fine-tuning of large models on low-end GPUs by optimizing SSD-CPU communication and computation, achieving higher GPU utilization compared to ZeRO-Infinity.
ZeRO-Infinity Fuyou GPU utilization SSD-CPU communication computation large models fine-tuning GPT-3 TFLOPS RTX 4090