Scaling Law for Transfer Learning. While the scaling behavior of pretrained LLMs has been extensively studied and
exhibits clear predictability, it becomes less clear when predicting the performance of pretrained LLMs on downstream tasks.
The work in [102] investigates the scaling behavior when fine-tuning a pretrained model and demonstrates that favorable
scaling laws, akin to those in [124], apply to transfer and few-shot settings in NLP tasks. In comparison to models trained
from scratch, pretrained models exhibit a more advantageous scaling law in low-data scenarios. Different scaling behaviors
between upstream and downstream configurations are observed in [262], indicating that, in addition to model size, the model’s
architecture plays a critical role in downstream fine-tuning. Specifically, the study demonstrates that redesigned models can
attain similar downstream fine-tuning quality while having 50% fewer parameters and training 40% faster when compared to
TheEfficiencySpectrumofLargeLanguageModels-AnAlgorithmicSurvey