title: Pre-Training to Learn in Context
summary: In-context learning, where pre-trained language models learn to perform tasks
  from task examples and instructions in their contexts, has attracted much
  attention in the NLP community. However, the ability of in-context learning is
  not fully exploited because language models are not explicitly trained to learn
  in context. To this end, we propose PICL (Pre-training for In-Context
  Learning), a framework to enhance the language models' in-context learning
  ability by pre-training the model on a large collection of "intrinsic tasks" in
  the general plain-text corpus using the simple language modeling objective.
  PICL encourages the model to infer and perform tasks by conditioning on the
  contexts while maintaining task generalization of pre-trained models. We
  evaluate the in-context learning performance of the model trained with PICL on
  seven widely-used text classification datasets and the Super-NaturalInstrctions
  benchmark, which contains 100+ NLP tasks formulated to text generation. Our
  experiments show that PICL is more effective and task-generalizable than a
  range of baselines, outperforming larger language models with nearly 4x
  parameters. The code is publicly available at https://github.com/thu-coai/PICL.
paper_authors._id: 64642753e0475a83333dacec
paper_authors.name: Yuxian Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T10:28:05.554Z
paper_authors._id: 624ac662102fcdff87be51b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/624ac662102fcdff87be51b9/rzNahZFFkp194170tactJ.jpeg
paper_authors.fullname: Yuxian Gu
paper_authors.type: user
paper_authors.user: t1101675
paper_authors._id: 64642753e0475a83333daced
paper_authors.name: Li Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T15:20:26.184Z
paper_authors._id: 5df85abada6d0311fd3d5408
paper_authors.avatarUrl: /avatars/2331cf703c1b5d3a62e2050b1a6eb108.svg
paper_authors.fullname: Li Dong
paper_authors.type: user
paper_authors.user: unilm
paper_authors._id: 64642753e0475a83333dacee
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:29:27.910Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_authors._id: 64642753e0475a83333dacef
paper_authors.name: Minlie Huang
paper_title: Pre-Training to Learn in Context
paper_summary: In-context learning, where pre-trained language models learn to perform tasks
  from task examples and instructions in their contexts, has attracted much
  attention in the NLP community. However, the ability of in-context learning is
  not fully exploited because language models are not explicitly trained to learn
  in context. To this end, we propose PICL (Pre-training for In-Context
  Learning), a framework to enhance the language models' in-context learning
  ability by pre-training the model on a large collection of "intrinsic tasks" in
  the general plain-text corpus using the simple language modeling objective.
  PICL encourages the model to infer and perform tasks by conditioning on the
  contexts while maintaining task generalization of pre-trained models. We
  evaluate the in-context learning performance of the model trained with PICL on
  seven widely-used text classification datasets and the Super-NaturalInstrctions
  benchmark, which contains 100+ NLP tasks formulated to text generation. Our
  experiments show that PICL is more effective and task-generalizable than a
  range of baselines, outperforming larger language models with nearly 4x
  parameters. The code is publicly available at https://github.com/thu-coai/PICL.
paper_ai_keywords: in-context learning pre-trained language models intrinsic tasks plain-text reasoning language modeling objective task generalization text classification Super-NaturalInstructions benchmark text generation