Continual pre-training Continual pre-training consists in performing another round of pre-training
from a pre-trained LLM, typically with a lesser volume of data than in the first stage. Such process
may be useful to quickly adapt to a new domain or elicit new properties in the LLM. For instance,
continual pre-training is used in Lemur (Xu et al., 2023d) to improve coding and reasoning capacities,
and Llama-2-long (Xiong et al., 2023) to extend context window.
ChatGPT’sOne-yearAnniversary-AreOpen-Source LargeLanguageModelsCatchingup