title: Amuro & Char: Analyzing the Relationship between Pre-Training and
    Fine-Tuning of Large Language Models
summary: The development of large language models leads to the formation of a
  pre-train-then-align paradigm, in which the model is typically pre-trained on a
  large text corpus and undergoes a tuning stage to align the model with human
  preference or downstream tasks. In this work, we investigate the relationship
  between pre-training and fine-tuning by fine-tuning multiple intermediate
  pre-trained model checkpoints. Our results on 18 datasets suggest that i)
  continual pre-training improves the model in a latent way that unveils after
  fine-tuning; ii) with extra fine-tuning, the datasets that the model does not
  demonstrate capability gain much more than those that the model performs well
  during the pre-training stage; iii) although model benefits significantly
  through supervised fine-tuning, it may forget previously known domain knowledge
  and the tasks that are not seen during fine-tuning; iv) the model resembles
  high sensitivity to evaluation prompts after supervised fine-tuning, but this
  sensitivity can be alleviated by more pre-training.
paper_authors._id: 66bc84a33ae330ae8bb70232
paper_authors.name: Kaiser Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T11:09:49.346Z
paper_authors._id: 616745f893f6837511d9ae25
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616745f893f6837511d9ae25/H0JIGDL5VOu29zVd4KiKB.jpeg
paper_authors.fullname: Kaiser Sun
paper_authors.type: user
paper_authors.user: KaiserWhoLearns
paper_authors._id: 66bc84a33ae330ae8bb70233
paper_authors.name: Mark Dredze
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-14T11:10:26.376Z
paper_authors._id: 6140f23f6dc32b0812420fc5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631646255954-noauth.jpeg
paper_authors.fullname: Mark Dredze
paper_authors.type: user
paper_authors.user: mdredze
paper_title: Amuro & Char: Analyzing the Relationship between Pre-Training and
    Fine-Tuning of Large Language Models
paper_summary: The development of large language models leads to the formation of a
  pre-train-then-align paradigm, in which the model is typically pre-trained on a
  large text corpus and undergoes a tuning stage to align the model with human
  preference or downstream tasks. In this work, we investigate the relationship
  between pre-training and fine-tuning by fine-tuning multiple intermediate
  pre-trained model checkpoints. Our results on 18 datasets suggest that i)
  continual pre-training improves the model in a latent way that unveils after
  fine-tuning; ii) with extra fine-tuning, the datasets that the model does not
  demonstrate capability gain much more than those that the model performs well
  during the pre-training stage; iii) although model benefits significantly
  through supervised fine-tuning, it may forget previously known domain knowledge
  and the tasks that are not seen during fine-tuning; iv) the model resembles
  high sensitivity to evaluation prompts after supervised fine-tuning, but this
  sensitivity can be alleviated by more pre-training.
paper_ai_keywords: large language models pre-train-then-align paradigm pre-training fine-tuning intermediate pre-trained model checkpoints latent way continual pre-training domain knowledge evaluation prompts