title: Adapting Large Language Models via Reading Comprehension
summary: We explore how continued pre-training on domain-specific corpora influences
  large language models, revealing that training on the raw corpora endows the
  model with domain knowledge, but drastically hurts its prompting ability for
  question answering. Taken inspiration from human learning via reading
  comprehension--practice after reading improves the ability to answer questions
  based on the learned knowledge--we propose a simple method for transforming raw
  corpora into reading comprehension texts. Each raw text is enriched with a
  series of tasks related to its content. Our method, highly scalable and
  applicable to any pre-training corpora, consistently enhances performance
  across various tasks in three different domains: biomedicine, finance, and law.
  Notably, our 7B language model achieves competitive performance with
  domain-specific models of much larger scales, such as BloombergGPT-50B.
  Furthermore, we demonstrate that domain-specific reading comprehension texts
  can improve the model's performance even on general benchmarks, showing the
  potential to develop a general model across even more domains. Our model, code,
  and data will be available at https://github.com/microsoft/LMOps.
paper_authors._id: 650912707e0d56c27157af44
paper_authors.name: Daixuan Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-19T15:54:06.813Z
paper_authors._id: 649e6761f9134a06ed1e0cea
paper_authors.avatarUrl: /avatars/00b5dcb744c54a4aa18fe08efd70d6ff.svg
paper_authors.fullname: Daixuan Cheng
paper_authors.type: user
paper_authors.user: daixuancheng
paper_authors._id: 650912707e0d56c27157af45
paper_authors.name: Shaohan Huang
paper_authors._id: 650912707e0d56c27157af46
paper_authors.name: Furu Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T13:13:58.991Z
paper_authors._id: 6368c512fbfe97c16a40baba
paper_authors.avatarUrl: /avatars/1c23bc7c0b6d9225699ce27647623d7a.svg
paper_authors.fullname: Furu Wei
paper_authors.type: user
paper_authors.user: thegenerality
paper_title: Adapting Large Language Models via Reading Comprehension
paper_summary: We explore how continued pre-training on domain-specific corpora influences
  large language models, revealing that training on the raw corpora endows the
  model with domain knowledge, but drastically hurts its prompting ability for
  question answering. Taken inspiration from human learning via reading
  comprehension--practice after reading improves the ability to answer questions
  based on the learned knowledge--we propose a simple method for transforming raw
  corpora into reading comprehension texts. Each raw text is enriched with a
  series of tasks related to its content. Our method, highly scalable and
  applicable to any pre-training corpora, consistently enhances performance
  across various tasks in three different domains: biomedicine, finance, and law.
  Notably, our 7B language model achieves competitive performance with
  domain-specific models of much larger scales, such as BloombergGPT-50B.
  Furthermore, we demonstrate that domain-specific reading comprehension texts
  can improve the model's performance even on general benchmarks, showing the
  potential to develop a general model across even more domains. Our model, code,
  and data will be available at https://github.com/microsoft/LMOps.
paper_ai_keywords: large language models domain-specific corpora reading comprehension texts stephen