title: How Do Large Language Models Acquire Factual Knowledge During
    Pretraining?
summary: Despite the recent observation that large language models (LLMs) can store
  substantial factual knowledge, there is a limited understanding of the
  mechanisms of how they acquire factual knowledge through pretraining. This work
  addresses this gap by studying how LLMs acquire factual knowledge during
  pretraining. The findings reveal several important insights into the dynamics
  of factual knowledge acquisition during pretraining. First, counterintuitively,
  we observe that pretraining on more data shows no significant improvement in
  the model's capability to acquire and maintain factual knowledge. Next, there
  is a power-law relationship between training steps and forgetting of
  memorization and generalization of factual knowledge, and LLMs trained with
  duplicated training data exhibit faster forgetting. Third, training LLMs with
  larger batch sizes can enhance the models' robustness to forgetting. Overall,
  our observations suggest that factual knowledge acquisition in LLM pretraining
  occurs by progressively increasing the probability of factual knowledge
  presented in the pretraining data at each step. However, this increase is
  diluted by subsequent forgetting. Based on this interpretation, we demonstrate
  that we can provide plausible explanations for recently observed behaviors of
  LLMs, such as the poor performance of LLMs on long-tail knowledge and the
  benefits of deduplicating the pretraining corpus.
paper_authors._id: 66715c42a3f8ed647dd1274d
paper_authors.name: Hoyeon Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:27:15.869Z
paper_authors._id: 64d0d6684dfd5df70744b237
paper_authors.avatarUrl: /avatars/4ea57bfd407e8cb727c624f64af75478.svg
paper_authors.fullname: Chang
paper_authors.type: user
paper_authors.user: Hoyeon
paper_authors._id: 66715c42a3f8ed647dd1274e
paper_authors.name: Jinho Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:27:23.995Z
paper_authors._id: 638467ee8283412d401770dd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638467ee8283412d401770dd/UAMwDHhSwf91XSsubfrS_.jpeg
paper_authors.fullname: Jinho Park
paper_authors.type: user
paper_authors.user: Br3ad
paper_authors._id: 66715c42a3f8ed647dd1274f
paper_authors.name: Seonghyeon Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:55:41.966Z
paper_authors._id: 62551f7767f0b85962624047
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664552038624-62551f7767f0b85962624047.png
paper_authors.fullname: Seonghyeon Ye
paper_authors.type: user
paper_authors.user: seonghyeonye
paper_authors._id: 66715c42a3f8ed647dd12750
paper_authors.name: Sohee Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:55:48.378Z
paper_authors._id: 606ae0adb579bb0e88515311
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1617617060380-noauth.jpeg
paper_authors.fullname: Sohee Yang
paper_authors.type: user
paper_authors.user: soheeyang
paper_authors._id: 66715c42a3f8ed647dd12751
paper_authors.name: Youngkyung Seo
paper_authors._id: 66715c42a3f8ed647dd12752
paper_authors.name: Du-Seong Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:56:07.197Z
paper_authors._id: 653b44214d105d9696cf5e7a
paper_authors.avatarUrl: /avatars/907509576817ec2199a07d9b42af9c42.svg
paper_authors.fullname: Du-Seong Chang
paper_authors.type: user
paper_authors.user: Du-Seong
paper_authors._id: 66715c42a3f8ed647dd12753
paper_authors.name: Minjoon Seo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T14:56:53.467Z
paper_authors._id: 621f05ba970615ad5861ceb1
paper_authors.avatarUrl: /avatars/7e1902aa71369a524afda9b0a9e88e22.svg
paper_authors.fullname: Minjoon Seo
paper_authors.type: user
paper_authors.user: minjoon
paper_title: How Do Large Language Models Acquire Factual Knowledge During
    Pretraining?
paper_summary: Despite the recent observation that large language models (LLMs) can store
  substantial factual knowledge, there is a limited understanding of the
  mechanisms of how they acquire factual knowledge through pretraining. This work
  addresses this gap by studying how LLMs acquire factual knowledge during
  pretraining. The findings reveal several important insights into the dynamics
  of factual knowledge acquisition during pretraining. First, counterintuitively,
  we observe that pretraining on more data shows no significant improvement in
  the model's capability to acquire and maintain factual knowledge. Next, there
  is a power-law relationship between training steps and forgetting of
  memorization and generalization of factual knowledge, and LLMs trained with
  duplicated training data exhibit faster forgetting. Third, training LLMs with
  larger batch sizes can enhance the models' robustness to forgetting. Overall,
  our observations suggest that factual knowledge acquisition in LLM pretraining
  occurs by progressively increasing the probability of factual knowledge
  presented in the pretraining data at each step. However, this increase is
  diluted by subsequent forgetting. Based on this interpretation, we demonstrate
  that we can provide plausible explanations for recently observed behaviors of
  LLMs, such as the poor performance of LLMs on long-tail knowledge and the
  benefits of deduplicating the pretraining corpus.
paper_ai_summary: LLMs acquire factual knowledge during pretraining through a process involving memorization, generalization, and forgetting, which is influenced by factors like data volume, batch size, and data duplication.
paper_ai_keywords: large language models (LLMs) factual knowledge acquisition pretraining memorization generalization forgetting power-law relationship batch size data duplication long-tail knowledge deduplicating the pretraining corpus