title: When Scaling Meets LLM Finetuning: The Effect of Data, Model and
    Finetuning Method
summary: While large language models (LLMs) often adopt finetuning to unlock their
  capabilities for downstream applications, our understanding on the inductive
  biases (especially the scaling properties) of different finetuning methods is
  still limited. To fill this gap, we conduct systematic experiments studying
  whether and how different scaling factors, including LLM model size,
  pretraining data size, new finetuning parameter size and finetuning data size,
  affect the finetuning performance. We consider two types of finetuning --
  full-model tuning (FMT) and parameter efficient tuning (PET, including prompt
  tuning and LoRA), and explore their scaling behaviors in the data-limited
  regime where the LLM model size substantially outweighs the finetuning data
  size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and
  experiments on bilingual machine translation and multilingual summarization
  benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative
  joint scaling law between finetuning data size and each other scaling factor;
  2) LLM finetuning benefits more from LLM model scaling than pretraining data
  scaling, and PET parameter scaling is generally ineffective; and 3) the optimal
  finetuning method is highly task- and finetuning data-dependent. We hope our
  findings could shed light on understanding, selecting and developing LLM
  finetuning methods.
paper_authors._id: 65ded0aaeb859000d9db2e47
paper_authors.name: Biao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:41:10.554Z
paper_authors._id: 6346a600bcb5e67902ac6f44
paper_authors.avatarUrl: /avatars/cf42043c19990bb388223a61c7a1e292.svg
paper_authors.fullname: Biao Zhang
paper_authors.type: user
paper_authors.user: bzhanggo
paper_authors._id: 65ded0aaeb859000d9db2e48
paper_authors.name: Zhongtao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:41:18.893Z
paper_authors._id: 63da416a3b8591bd11f4d09d
paper_authors.avatarUrl: /avatars/5af13490d6d2a0e035a4f9ba87381658.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: Zhongtao
paper_authors._id: 65ded0aaeb859000d9db2e49
paper_authors.name: Colin Cherry
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-28T12:41:25.824Z
paper_authors._id: 6144bb22207b8aba44e8ec59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631896530912-6144bb22207b8aba44e8ec59.jpeg
paper_authors.fullname: Colin Cherry
paper_authors.type: user
paper_authors.user: ColinCherry
paper_authors._id: 65ded0aaeb859000d9db2e4a
paper_authors.name: Orhan Firat
paper_title: When Scaling Meets LLM Finetuning: The Effect of Data, Model and
    Finetuning Method
paper_summary: While large language models (LLMs) often adopt finetuning to unlock their
  capabilities for downstream applications, our understanding on the inductive
  biases (especially the scaling properties) of different finetuning methods is
  still limited. To fill this gap, we conduct systematic experiments studying
  whether and how different scaling factors, including LLM model size,
  pretraining data size, new finetuning parameter size and finetuning data size,
  affect the finetuning performance. We consider two types of finetuning --
  full-model tuning (FMT) and parameter efficient tuning (PET, including prompt
  tuning and LoRA), and explore their scaling behaviors in the data-limited
  regime where the LLM model size substantially outweighs the finetuning data
  size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and
  experiments on bilingual machine translation and multilingual summarization
  benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative
  joint scaling law between finetuning data size and each other scaling factor;
  2) LLM finetuning benefits more from LLM model scaling than pretraining data
  scaling, and PET parameter scaling is generally ineffective; and 3) the optimal
  finetuning method is highly task- and finetuning data-dependent. We hope our
  findings could shed light on understanding, selecting and developing LLM
  finetuning methods.
paper_ai_keywords: inductive biases large language models (LLMs) systematic experiments finetuning methods scaling factors LLM model size pretraining data size finetuning parameter size finetuning data size finetuning performance full-model tuning (FMT) parameter efficient tuning (PET) prompt tuning LoRA data-limited regime bilingual LLMs bilingual machine translation multilingual summarization joint scaling law LLM model scaling PET parameter scaling optimal finetuning method task-dependent finetuning data-dependent