title: Instruction Pre-Training: Language Models are Supervised Multitask
    Learners
summary: Unsupervised multitask pre-training has been the critical method behind the
  recent success of language models (LMs). However, supervised multitask learning
  still holds significant promise, as scaling it in the post-training stage
  trends towards better generalization. In this paper, we explore supervised
  multitask pre-training by proposing Instruction Pre-Training, a framework that
  scalably augments massive raw corpora with instruction-response pairs to
  pre-train LMs. The instruction-response pairs are generated by an efficient
  instruction synthesizer built on open-source models. In our experiments, we
  synthesize 200M instruction-response pairs covering 40+ task categories to
  verify the effectiveness of Instruction Pre-Training. In pre-training from
  scratch, Instruction Pre-Training not only consistently enhances pre-trained
  base models but also benefits more from further instruction tuning. In
  continual pre-training, Instruction Pre-Training enables Llama3-8B to be
  comparable to or even outperform Llama3-70B. Our model, code, and data are
  available at https://github.com/microsoft/LMOps.
paper_authors._id: 6674ef905f7d5c8af70b5607
paper_authors.name: Daixuan Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:05:43.020Z
paper_authors._id: 649e6761f9134a06ed1e0cea
paper_authors.avatarUrl: /avatars/00b5dcb744c54a4aa18fe08efd70d6ff.svg
paper_authors.fullname: Daixuan Cheng
paper_authors.type: user
paper_authors.user: daixuancheng
paper_authors._id: 6674ef905f7d5c8af70b5608
paper_authors.name: Yuxian Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:12:42.585Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 6674ef905f7d5c8af70b5609
paper_authors.name: Shaohan Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:12:55.699Z
paper_authors._id: 632bd2f72d6a805eeb4bc601
paper_authors.avatarUrl: /avatars/6e1533e8a599f3068290aa69ac82cab7.svg
paper_authors.fullname: HUANG SHAOHAN
paper_authors.type: user
paper_authors.user: buaahsh
paper_authors._id: 6674ef905f7d5c8af70b560a
paper_authors.name: Junyu Bi
paper_authors._id: 6674ef905f7d5c8af70b560b
paper_authors.name: Minlie Huang
paper_authors._id: 6674ef905f7d5c8af70b560c
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T11:13:45.390Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Instruction Pre-Training: Language Models are Supervised Multitask
    Learners
paper_summary: Unsupervised multitask pre-training has been the critical method behind the
  recent success of language models (LMs). However, supervised multitask learning
  still holds significant promise, as scaling it in the post-training stage
  trends towards better generalization. In this paper, we explore supervised
  multitask pre-training by proposing Instruction Pre-Training, a framework that
  scalably augments massive raw corpora with instruction-response pairs to
  pre-train LMs. The instruction-response pairs are generated by an efficient
  instruction synthesizer built on open-source models. In our experiments, we
  synthesize 200M instruction-response pairs covering 40+ task categories to
  verify the effectiveness of Instruction Pre-Training. In pre-training from
  scratch, Instruction Pre-Training not only consistently enhances pre-trained
  base models but also benefits more from further instruction tuning. In
  continual pre-training, Instruction Pre-Training enables Llama3-8B to be
  comparable to or even outperform Llama3-70B. Our model, code, and data are
  available at https://github.com/microsoft/LMOps.
paper_ai_summary: Instruction Pre-Training enhances language models by generating and incorporating instruction-response pairs into unsupervised multitask pre-training.
paper_ai_keywords: Instruction Pre-Training instruction-response pairs instruction synthesizer open-source models pre-training continual pre-training Llama3-8B Llama3-70B