How Do Large Language Models Acquire Factual Knowledge During
  Pretraining?
Despite the recent observation that large language models (LLMs) can store
substantial factual knowledge, there is a limited understanding of the
mechanisms of how they acquire factual knowledge through pretraining. This work
addresses this gap by studying how LLMs acquire factual knowledge during
pretraining. The findings reveal several important insights into the dynamics
of factual knowledge acquisition during pretraining. First, counterintuitively,
we observe that pretraining on more data shows no significant improvement in
the model's capability to acquire and maintain factual knowledge. Next, there
is a power-law relationship between training steps and forgetting of
memorization and generalization of factual knowledge, and LLMs trained with
duplicated training data exhibit faster forgetting. Third, training LLMs with
larger batch sizes can enhance the models' robustness to forgetting. Overall,
our observations suggest that factual knowledge acquisition in LLM pretraining
occurs by progressively increasing the probability of factual knowledge
presented in the pretraining data at each step. However, this increase is
diluted by subsequent forgetting. Based on this interpretation, we demonstrate
that we can provide plausible explanations for recently observed behaviors of
LLMs, such as the poor performance of LLMs on long-tail knowledge and the
benefits of deduplicating the pretraining corpus.
66715c42a3f8ed647dd1274d
Hoyeon Chang
admin_assigned
2024-06-18T14:27:15.869Z
64d0d6684dfd5df70744b237
/avatars/4ea57bfd407e8cb727c624f64af75478.svg
Chang
user
Hoyeon
66715c42a3f8ed647dd1274e
Jinho Park
admin_assigned
2024-06-18T14:27:23.995Z
638467ee8283412d401770dd
https://cdn-avatars.huggingface.co/v1/production/uploads/638467ee8283412d401770dd/UAMwDHhSwf91XSsubfrS_.jpeg
Jinho Park
user
Br3ad
66715c42a3f8ed647dd1274f
Seonghyeon Ye
admin_assigned
2024-06-18T14:55:41.966Z
62551f7767f0b85962624047
https://cdn-avatars.huggingface.co/v1/production/uploads/1664552038624-62551f7767f0b85962624047.png
Seonghyeon Ye
user
seonghyeonye
66715c42a3f8ed647dd12750
Sohee Yang
admin_assigned
2024-06-18T14:55:48.378Z
606ae0adb579bb0e88515311
https://cdn-avatars.huggingface.co/v1/production/uploads/1617617060380-noauth.jpeg
Sohee Yang
user
soheeyang
66715c42a3f8ed647dd12751
Youngkyung Seo
66715c42a3f8ed647dd12752
Du-Seong Chang
admin_assigned
2024-06-18T14:56:07.197Z
653b44214d105d9696cf5e7a
/avatars/907509576817ec2199a07d9b42af9c42.svg
Du-Seong Chang
user
Du-Seong
66715c42a3f8ed647dd12753
Minjoon Seo
admin_assigned
2024-06-18T14:56:53.467Z
621f05ba970615ad5861ceb1
/avatars/7e1902aa71369a524afda9b0a9e88e22.svg
Minjoon Seo
user
minjoon
How Do Large Language Models Acquire Factual Knowledge During
  Pretraining?
Despite the recent observation that large language models (LLMs) can store
substantial factual knowledge, there is a limited understanding of the
mechanisms of how they acquire factual knowledge through pretraining. This work
addresses this gap by studying how LLMs acquire factual knowledge during
pretraining. The findings reveal several important insights into the dynamics
of factual knowledge acquisition during pretraining. First, counterintuitively,
we observe that pretraining on more data shows no significant improvement in
the model's capability to acquire and maintain factual knowledge. Next, there
is a power-law relationship between training steps and forgetting of
memorization and generalization of factual knowledge, and LLMs trained with
duplicated training data exhibit faster forgetting. Third, training LLMs with
larger batch sizes can enhance the models' robustness to forgetting. Overall,
our observations suggest that factual knowledge acquisition in LLM pretraining
occurs by progressively increasing the probability of factual knowledge
presented in the pretraining data at each step. However, this increase is
diluted by subsequent forgetting. Based on this interpretation, we demonstrate
that we can provide plausible explanations for recently observed behaviors of
LLMs, such as the poor performance of LLMs on long-tail knowledge and the
benefits of deduplicating the pretraining corpus.
LLMs acquire factual knowledge during pretraining through a process involving memorization, generalization, and forgetting, which is influenced by factors like data volume, batch size, and data duplication.
large language models (LLMs) factual knowledge acquisition pretraining memorization generalization forgetting power-law relationship batch size data duplication long-tail knowledge deduplicating the pretraining corpus