title: How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on
    Continual Pre-Training
summary: Despite exceptional capabilities in knowledge-intensive tasks, Large Language
  Models (LLMs) face a critical gap in understanding how they internalize new
  knowledge, particularly how to structurally embed acquired knowledge in their
  neural computations. We address this issue through the lens of knowledge
  circuit evolution, identifying computational subgraphs that facilitate
  knowledge storage and processing. Our systematic analysis of circuit evolution
  throughout continual pre-training reveals several key findings: (1) the
  acquisition of new knowledge is influenced by its relevance to pre-existing
  knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase
  shift from formation to optimization; (3) the evolution of knowledge circuits
  follows a deep-to-shallow pattern. These insights not only advance our
  theoretical understanding of the mechanisms of new knowledge acquisition in
  LLMs, but also provide potential implications for improving continual
  pre-training strategies to enhance model performance. Code and data will be
  available at https://github.com/zjunlp/DynamicKnowledgeCircuits.
paper_authors._id: 67b42223c2fe54b8d43efed6
paper_authors.name: Yixin Ou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:22:40.840Z
paper_authors._id: 6241749cf80bd930bd99f3dd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669210243382-6241749cf80bd930bd99f3dd.jpeg
paper_authors.fullname: Ou Yixin
paper_authors.type: user
paper_authors.user: OE-Heart
paper_authors._id: 67b42223c2fe54b8d43efed7
paper_authors.name: Yunzhi Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:22:56.226Z
paper_authors._id: 6122fbe636a4c36a99dbea7b
paper_authors.avatarUrl: /avatars/c0cd2c1ef58e315d9adda9d26000f625.svg
paper_authors.fullname: Yunzhi Yao
paper_authors.type: user
paper_authors.user: cowTodd
paper_authors._id: 67b42223c2fe54b8d43efed8
paper_authors.name: Ningyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:31:04.227Z
paper_authors._id: 620b3bbb0668e435407c8d0a
paper_authors.avatarUrl: /avatars/e0fccbb2577d76088e09f054c35cffbc.svg
paper_authors.fullname: Ningyu Zhang
paper_authors.type: user
paper_authors.user: Ningyu
paper_authors._id: 67b42223c2fe54b8d43efed9
paper_authors.name: Hui Jin
paper_authors._id: 67b42223c2fe54b8d43efeda
paper_authors.name: Jiacheng Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:23:08.677Z
paper_authors._id: 6591895250d39af7f431ad3d
paper_authors.avatarUrl: /avatars/83c9717e7d8bea8cae6a640ee6455214.svg
paper_authors.fullname: Sun Jiacheng
paper_authors.type: user
paper_authors.user: sunjc826
paper_authors._id: 67b42223c2fe54b8d43efedb
paper_authors.name: Shumin Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:23:17.528Z
paper_authors._id: 6441f1d2603214724ec0c1c2
paper_authors.avatarUrl: /avatars/d3c4b759e6a5635e37ff715fae52e5ba.svg
paper_authors.fullname: Shumin Deng
paper_authors.type: user
paper_authors.user: 231sm
paper_authors._id: 67b42223c2fe54b8d43efedc
paper_authors.name: Zhenguo Li
paper_authors._id: 67b42223c2fe54b8d43efedd
paper_authors.name: Huajun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:23:34.297Z
paper_authors._id: 64931296137833d7ec7689cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64931296137833d7ec7689cd/TBihNdp1ZwIWjhfAWjRr6.jpeg
paper_authors.fullname: Huajun Chen
paper_authors.type: user
paper_authors.user: huajunsir
paper_title: How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on
    Continual Pre-Training
paper_summary: Despite exceptional capabilities in knowledge-intensive tasks, Large Language
  Models (LLMs) face a critical gap in understanding how they internalize new
  knowledge, particularly how to structurally embed acquired knowledge in their
  neural computations. We address this issue through the lens of knowledge
  circuit evolution, identifying computational subgraphs that facilitate
  knowledge storage and processing. Our systematic analysis of circuit evolution
  throughout continual pre-training reveals several key findings: (1) the
  acquisition of new knowledge is influenced by its relevance to pre-existing
  knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase
  shift from formation to optimization; (3) the evolution of knowledge circuits
  follows a deep-to-shallow pattern. These insights not only advance our
  theoretical understanding of the mechanisms of new knowledge acquisition in
  LLMs, but also provide potential implications for improving continual
  pre-training strategies to enhance model performance. Code and data will be
  available at https://github.com/zjunlp/DynamicKnowledgeCircuits.
paper_ai_keywords: Large Language Models (LLMs) knowledge circuit evolution computational subgraphs knowledge storage knowledge processing continual pre-training phase shift deep-to-shallow pattern