Scaling Laws for Downstream Task Performance of Large Language Models
Scaling laws provide important insights that can guide the design of large
language models (LLMs). Existing work has primarily focused on studying scaling
laws for pretraining (upstream) loss. However, in transfer learning settings,
in which LLMs are pretrained on an unsupervised dataset and then finetuned on a
downstream task, we often also care about the downstream performance. In this
work, we study the scaling behavior in a transfer learning setting, where LLMs
are finetuned for machine translation tasks. Specifically, we investigate how
the choice of the pretraining data and its size affect downstream performance
(translation quality) as judged by two metrics: downstream cross-entropy and
BLEU score. Our experiments indicate that the size of the finetuning dataset
and the distribution alignment between the pretraining and downstream data
significantly influence the scaling behavior. With sufficient alignment, both
downstream cross-entropy and BLEU score improve monotonically with more
pretraining data. In such cases, we show that it is possible to predict the
downstream BLEU score with good accuracy using a log-law. However, there are
also cases where moderate misalignment causes the BLEU score to fluctuate or
get worse with more pretraining, whereas downstream cross-entropy monotonically
improves. By analyzing these observations, we provide new practical insights
for choosing appropriate pretraining data.
65c2f609477256c85ea4a78b
Berivan Isik
claimed_verified
2024-02-08T09:50:18.408Z
649af6e90631e151968e3926
/avatars/3fe27659cba3211f4c04e395f844c796.svg
Berivan Isik
user
berivan
65c2f609477256c85ea4a78c
Natalia Ponomareva
65c2f609477256c85ea4a78d
Hussein Hazimeh
65c2f609477256c85ea4a78e
Dimitris Paparas
65c2f609477256c85ea4a78f
Sergei Vassilvitskii
65c2f609477256c85ea4a790
Sanmi Koyejo
claimed_verified
2024-04-03T09:04:11.853Z
64931e7e2da595588288f161
https://cdn-avatars.huggingface.co/v1/production/uploads/64931e7e2da595588288f161/4jOhJOFsU7RVFMgGk5kO7.jpeg
Sanmi Koyejo
user
sanmikoyejo
Scaling Laws for Downstream Task Performance of Large Language Models
Scaling laws provide important insights that can guide the design of large
language models (LLMs). Existing work has primarily focused on studying scaling
laws for pretraining (upstream) loss. However, in transfer learning settings,
in which LLMs are pretrained on an unsupervised dataset and then finetuned on a
downstream task, we often also care about the downstream performance. In this
work, we study the scaling behavior in a transfer learning setting, where LLMs
are finetuned for machine translation tasks. Specifically, we investigate how
the choice of the pretraining data and its size affect downstream performance
(translation quality) as judged by two metrics: downstream cross-entropy and
BLEU score. Our experiments indicate that the size of the finetuning dataset
and the distribution alignment between the pretraining and downstream data
significantly influence the scaling behavior. With sufficient alignment, both
downstream cross-entropy and BLEU score improve monotonically with more
pretraining data. In such cases, we show that it is possible to predict the
downstream BLEU score with good accuracy using a log-law. However, there are
also cases where moderate misalignment causes the BLEU score to fluctuate or
get worse with more pretraining, whereas downstream cross-entropy monotonically
improves. By analyzing these observations, we provide new practical insights
for choosing appropriate pretraining data.
In transfer learning for machine translation, the size and alignment of pretraining data significantly impact downstream performance as measured by cross-entropy and BLEU score.
large language models scaling laws pretraining finetuning machine translation downstream performance pretraining data downstream cross-entropy BLEU score log-law data distribution alignment