title: Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to
    the Edge of Generalization
summary: We study whether transformers can learn to implicitly reason over parametric
  knowledge, a skill that even the most capable language models struggle with.
  Focusing on two representative reasoning types, composition and comparison, we
  consistently find that transformers can learn implicit reasoning, but only
  through grokking, i.e., extended training far beyond overfitting. The levels of
  generalization also vary across reasoning types: when faced with
  out-of-distribution examples, transformers fail to systematically generalize
  for composition but succeed for comparison. We delve into the model's internals
  throughout training, conducting analytical experiments that reveal: 1) the
  mechanism behind grokking, such as the formation of the generalizing circuit
  and its relation to the relative efficiency of generalizing and memorizing
  circuits, and 2) the connection between systematicity and the configuration of
  the generalizing circuit. Our findings guide data and training setup to better
  induce implicit reasoning and suggest potential improvements to the transformer
  architecture, such as encouraging cross-layer knowledge sharing. Furthermore,
  we demonstrate that for a challenging reasoning task with a large search space,
  GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly
  regardless of prompting styles or retrieval augmentation, while a fully grokked
  transformer can achieve near-perfect accuracy, showcasing the power of
  parametric memory for complex reasoning.
paper_authors._id: 6653e9e7f8137bb650b40fab
paper_authors.name: Boshi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T08:09:24.353Z
paper_authors._id: 62e5e742972081fe55828530
paper_authors.avatarUrl: /avatars/1cc9a4d5fbb634879a31f45bb7a5ec41.svg
paper_authors.fullname: Boshi Wang
paper_authors.type: user
paper_authors.user: boshiwang
paper_authors._id: 6653e9e7f8137bb650b40fac
paper_authors.name: Xiang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:18:12.062Z
paper_authors._id: 6230d750d93e84e233882dbc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
paper_authors.fullname: Xiang Yue
paper_authors.type: user
paper_authors.user: yuexiang96
paper_authors._id: 6653e9e7f8137bb650b40fad
paper_authors.name: Yu Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T19:21:59.248Z
paper_authors._id: 6477a323dbc2a416f8b852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6477a323dbc2a416f8b852b3/mRKW5kT9GASORT4YnaZz0.jpeg
paper_authors.fullname: Yu Su
paper_authors.type: user
paper_authors.user: ysu-nlp
paper_authors._id: 6653e9e7f8137bb650b40fae
paper_authors.name: Huan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-27T08:09:45.562Z
paper_authors._id: 6127b327699a5d2cde0b44dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629991702069-noauth.png
paper_authors.fullname: Huan Sun
paper_authors.type: user
paper_authors.user: huansun
paper_title: Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to
    the Edge of Generalization
paper_summary: We study whether transformers can learn to implicitly reason over parametric
  knowledge, a skill that even the most capable language models struggle with.
  Focusing on two representative reasoning types, composition and comparison, we
  consistently find that transformers can learn implicit reasoning, but only
  through grokking, i.e., extended training far beyond overfitting. The levels of
  generalization also vary across reasoning types: when faced with
  out-of-distribution examples, transformers fail to systematically generalize
  for composition but succeed for comparison. We delve into the model's internals
  throughout training, conducting analytical experiments that reveal: 1) the
  mechanism behind grokking, such as the formation of the generalizing circuit
  and its relation to the relative efficiency of generalizing and memorizing
  circuits, and 2) the connection between systematicity and the configuration of
  the generalizing circuit. Our findings guide data and training setup to better
  induce implicit reasoning and suggest potential improvements to the transformer
  architecture, such as encouraging cross-layer knowledge sharing. Furthermore,
  we demonstrate that for a challenging reasoning task with a large search space,
  GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly
  regardless of prompting styles or retrieval augmentation, while a fully grokked
  transformer can achieve near-perfect accuracy, showcasing the power of
  parametric memory for complex reasoning.
paper_ai_summary: Transformers can learn implicit reasoning through extensive training, with varying degrees of generalization for different reasoning types, and parametric memory proves more effective for complex reasoning tasks compared to non-parametric memory.
paper_ai_keywords: transformers implicit reasoning grokking overfitting composition comparison generalization generalizing circuit memorizing circuits systematicity cross-layer knowledge sharing parametric memory non-parametric memory GPT-4-Turbo Gemini-1.5-Pro