title: An Emulator for Fine-Tuning Large Language Models using Small Language
    Models
summary: Widely used language models (LMs) are typically built by scaling up a
  two-stage training pipeline: a pre-training stage that uses a very large,
  diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that
  uses targeted examples or other specifications of desired behaviors. While it
  has been hypothesized that knowledge and skills come from pre-training, and
  fine-tuning mostly filters this knowledge and skillset, this intuition has not
  been extensively tested. To aid in doing so, we introduce a novel technique for
  decoupling the knowledge and skills gained in these two stages, enabling a
  direct answer to the question, "What would happen if we combined the knowledge
  learned by a large model during pre-training with the knowledge learned by a
  small model during fine-tuning (or vice versa)?" Using an RL-based framework
  derived from recent developments in learning from human preferences, we
  introduce emulated fine-tuning (EFT), a principled and practical method for
  sampling from a distribution that approximates (or 'emulates') the result of
  pre-training and fine-tuning at different scales. Our experiments with EFT show
  that scaling up fine-tuning tends to improve helpfulness, while scaling up
  pre-training tends to improve factuality. Beyond decoupling scale, we show that
  EFT enables test-time adjustment of competing behavioral traits like
  helpfulness and harmlessness without additional training. Finally, a special
  case of emulated fine-tuning, which we call LM up-scaling, avoids
  resource-intensive fine-tuning of large pre-trained models by ensembling them
  with small fine-tuned models, essentially emulating the result of fine-tuning
  the large pre-trained model. Up-scaling consistently improves helpfulness and
  factuality of instruction-following models in the Llama, Llama-2, and Falcon
  families, without additional hyperparameters or training.
paper_authors._id: 65320dd29222842fbd18e321
paper_authors.name: Eric Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:51:20.238Z
paper_authors._id: 6514e9cbfc2301d62d7bf2ce
paper_authors.avatarUrl: /avatars/eba6bd6e2840d7b5dd735e6662020d0c.svg
paper_authors.fullname: Eric mitchell
paper_authors.type: user
paper_authors.user: Pain777
paper_authors._id: 65320dd29222842fbd18e322
paper_authors.name: Rafael Rafailov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:51:27.077Z
paper_authors._id: 64701c98d742e9ef6522598f
paper_authors.avatarUrl: /avatars/df3a52bd4bc374d7cb4061f449f75c07.svg
paper_authors.fullname: Rafael Rafailov
paper_authors.type: user
paper_authors.user: rmrafailov
paper_authors._id: 65320dd29222842fbd18e323
paper_authors.name: Archit Sharma
paper_authors._id: 65320dd29222842fbd18e324
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:52:02.811Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 65320dd29222842fbd18e325
paper_authors.name: Christopher D. Manning
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:52:37.883Z
paper_authors._id: 5f555a1f8bf55658acfed19e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628695407761-5f555a1f8bf55658acfed19e.jpeg
paper_authors.fullname: Christopher Manning
paper_authors.type: user
paper_authors.user: manning
paper_title: An Emulator for Fine-Tuning Large Language Models using Small Language
    Models
paper_summary: Widely used language models (LMs) are typically built by scaling up a
  two-stage training pipeline: a pre-training stage that uses a very large,
  diverse dataset of text and a fine-tuning (sometimes, 'alignment') stage that
  uses targeted examples or other specifications of desired behaviors. While it
  has been hypothesized that knowledge and skills come from pre-training, and
  fine-tuning mostly filters this knowledge and skillset, this intuition has not
  been extensively tested. To aid in doing so, we introduce a novel technique for
  decoupling the knowledge and skills gained in these two stages, enabling a
  direct answer to the question, "What would happen if we combined the knowledge
  learned by a large model during pre-training with the knowledge learned by a
  small model during fine-tuning (or vice versa)?" Using an RL-based framework
  derived from recent developments in learning from human preferences, we
  introduce emulated fine-tuning (EFT), a principled and practical method for
  sampling from a distribution that approximates (or 'emulates') the result of
  pre-training and fine-tuning at different scales. Our experiments with EFT show
  that scaling up fine-tuning tends to improve helpfulness, while scaling up
  pre-training tends to improve factuality. Beyond decoupling scale, we show that
  EFT enables test-time adjustment of competing behavioral traits like
  helpfulness and harmlessness without additional training. Finally, a special
  case of emulated fine-tuning, which we call LM up-scaling, avoids
  resource-intensive fine-tuning of large pre-trained models by ensembling them
  with small fine-tuned models, essentially emulating the result of fine-tuning
  the large pre-trained model. Up-scaling consistently improves helpfulness and
  factuality of instruction-following models in the Llama, Llama-2, and Falcon
  families, without additional hyperparameters or training.
paper_ai_keywords: emulated fine-tuning (EFT) learning from human preferences LM up-scaling ensemble instruction-following models Llama Llama-2 Falcon helpfulness factuality harmlessness