title: Scaling Laws for Downstream Task Performance of Large Language Models
summary: Scaling laws provide important insights that can guide the design of large
  language models (LLMs). Existing work has primarily focused on studying scaling
  laws for pretraining (upstream) loss. However, in transfer learning settings,
  in which LLMs are pretrained on an unsupervised dataset and then finetuned on a
  downstream task, we often also care about the downstream performance. In this
  work, we study the scaling behavior in a transfer learning setting, where LLMs
  are finetuned for machine translation tasks. Specifically, we investigate how
  the choice of the pretraining data and its size affect downstream performance
  (translation quality) as judged by two metrics: downstream cross-entropy and
  BLEU score. Our experiments indicate that the size of the finetuning dataset
  and the distribution alignment between the pretraining and downstream data
  significantly influence the scaling behavior. With sufficient alignment, both
  downstream cross-entropy and BLEU score improve monotonically with more
  pretraining data. In such cases, we show that it is possible to predict the
  downstream BLEU score with good accuracy using a log-law. However, there are
  also cases where moderate misalignment causes the BLEU score to fluctuate or
  get worse with more pretraining, whereas downstream cross-entropy monotonically
  improves. By analyzing these observations, we provide new practical insights
  for choosing appropriate pretraining data.
paper_authors._id: 65c2f609477256c85ea4a78b
paper_authors.name: Berivan Isik
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-08T09:50:18.408Z
paper_authors._id: 649af6e90631e151968e3926
paper_authors.avatarUrl: /avatars/3fe27659cba3211f4c04e395f844c796.svg
paper_authors.fullname: Berivan Isik
paper_authors.type: user
paper_authors.user: berivan
paper_authors._id: 65c2f609477256c85ea4a78c
paper_authors.name: Natalia Ponomareva
paper_authors._id: 65c2f609477256c85ea4a78d
paper_authors.name: Hussein Hazimeh
paper_authors._id: 65c2f609477256c85ea4a78e
paper_authors.name: Dimitris Paparas
paper_authors._id: 65c2f609477256c85ea4a78f
paper_authors.name: Sergei Vassilvitskii
paper_authors._id: 65c2f609477256c85ea4a790
paper_authors.name: Sanmi Koyejo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:04:11.853Z
paper_authors._id: 64931e7e2da595588288f161
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
paper_authors.fullname: Sanmi Koyejo
paper_authors.type: user
paper_authors.user: sanmikoyejo
paper_title: Scaling Laws for Downstream Task Performance of Large Language Models
paper_summary: Scaling laws provide important insights that can guide the design of large
  language models (LLMs). Existing work has primarily focused on studying scaling
  laws for pretraining (upstream) loss. However, in transfer learning settings,
  in which LLMs are pretrained on an unsupervised dataset and then finetuned on a
  downstream task, we often also care about the downstream performance. In this
  work, we study the scaling behavior in a transfer learning setting, where LLMs
  are finetuned for machine translation tasks. Specifically, we investigate how
  the choice of the pretraining data and its size affect downstream performance
  (translation quality) as judged by two metrics: downstream cross-entropy and
  BLEU score. Our experiments indicate that the size of the finetuning dataset
  and the distribution alignment between the pretraining and downstream data
  significantly influence the scaling behavior. With sufficient alignment, both
  downstream cross-entropy and BLEU score improve monotonically with more
  pretraining data. In such cases, we show that it is possible to predict the
  downstream BLEU score with good accuracy using a log-law. However, there are
  also cases where moderate misalignment causes the BLEU score to fluctuate or
  get worse with more pretraining, whereas downstream cross-entropy monotonically
  improves. By analyzing these observations, we provide new practical insights
  for choosing appropriate pretraining data.
paper_ai_summary: In transfer learning for machine translation, the size and alignment of pretraining data significantly impact downstream performance as measured by cross-entropy and BLEU score.
paper_ai_keywords: large language models scaling laws pretraining finetuning machine translation downstream performance pretraining data downstream cross-entropy BLEU score log-law data distribution alignment