also benefits large-scale generative models. We conduct initial experiments on three domains—
biomedicine, finance, and law—revealing that continued training on the raw corpora results in a
drastic drop in prompting performance but still benefits fine-tuning evaluation and knowledge prob-
ing tests. This leads us to conclude that domain-adaptive pre-training using raw corpora imparts
domain knowledge to the LLM while affecting its prompting ability.
To leverage domain-specific knowledge while enhancing prompting performance, we introduce a
simple method for transforming large-scale raw corpora into reading comprehension texts: each raw
text is enriched with a series of tasks relevant to its content, as illustrated in Figure 2. These tasks are
designed to help the model maintain its ability to answer questions using natural language, based on
the context of the raw text. Furthermore, we augment the reading comprehension texts with diverse
ADAPTINGLARGELANGUAGEMODELSVIA READINGCOMPREHENSION