When Scaling Meets LLM Finetuning: The Effect of Data, Model and
  Finetuning Method
While large language models (LLMs) often adopt finetuning to unlock their
capabilities for downstream applications, our understanding on the inductive
biases (especially the scaling properties) of different finetuning methods is
still limited. To fill this gap, we conduct systematic experiments studying
whether and how different scaling factors, including LLM model size,
pretraining data size, new finetuning parameter size and finetuning data size,
affect the finetuning performance. We consider two types of finetuning --
full-model tuning (FMT) and parameter efficient tuning (PET, including prompt
tuning and LoRA), and explore their scaling behaviors in the data-limited
regime where the LLM model size substantially outweighs the finetuning data
size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and
experiments on bilingual machine translation and multilingual summarization
benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative
joint scaling law between finetuning data size and each other scaling factor;
2) LLM finetuning benefits more from LLM model scaling than pretraining data
scaling, and PET parameter scaling is generally ineffective; and 3) the optimal
finetuning method is highly task- and finetuning data-dependent. We hope our
findings could shed light on understanding, selecting and developing LLM
finetuning methods.
65ded0aaeb859000d9db2e47
Biao Zhang
admin_assigned
2024-02-28T12:41:10.554Z
6346a600bcb5e67902ac6f44
https://cdn-avatars.huggingface.co/v1/production/uploads/6346a600bcb5e67902ac6f44/Kw-Ucmxx5S9WdlYDxQQ-z.jpeg
Biao Zhang
user
bzhanggo
65ded0aaeb859000d9db2e48
Zhongtao Liu
admin_assigned
2024-02-28T12:41:18.893Z
63da416a3b8591bd11f4d09d
/avatars/5af13490d6d2a0e035a4f9ba87381658.svg
Liu
user
Zhongtao
65ded0aaeb859000d9db2e49
Colin Cherry
admin_assigned
2024-02-28T12:41:25.824Z
6144bb22207b8aba44e8ec59
https://cdn-avatars.huggingface.co/v1/production/uploads/1631896530912-6144bb22207b8aba44e8ec59.jpeg
Colin Cherry
user
ColinCherry
65ded0aaeb859000d9db2e4a
Orhan Firat
When Scaling Meets LLM Finetuning: The Effect of Data, Model and
  Finetuning Method
While large language models (LLMs) often adopt finetuning to unlock their
capabilities for downstream applications, our understanding on the inductive
biases (especially the scaling properties) of different finetuning methods is
still limited. To fill this gap, we conduct systematic experiments studying
whether and how different scaling factors, including LLM model size,
pretraining data size, new finetuning parameter size and finetuning data size,
affect the finetuning performance. We consider two types of finetuning --
full-model tuning (FMT) and parameter efficient tuning (PET, including prompt
tuning and LoRA), and explore their scaling behaviors in the data-limited
regime where the LLM model size substantially outweighs the finetuning data
size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and
experiments on bilingual machine translation and multilingual summarization
benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative
joint scaling law between finetuning data size and each other scaling factor;
2) LLM finetuning benefits more from LLM model scaling than pretraining data
scaling, and PET parameter scaling is generally ineffective; and 3) the optimal
finetuning method is highly task- and finetuning data-dependent. We hope our
findings could shed light on understanding, selecting and developing LLM
finetuning methods.
Systematic experiments reveal scaling laws for large language model finetuning, showing that model size and finetuning data size are critical factors, while parameter-efficient tuning methods generally provide limited benefits.
finetuning inductive biases scaling properties full-model tuning parameter efficient tuning prompt tuning LoRA data-limited regime bilingual LLMs bilingual machine translation multilingual summarization power-based multiplicative joint scaling law task-dependent finetuning data-dependent