title: Grokking in the Wild: Data Augmentation for Real-World Multi-Hop
    Reasoning with Transformers
summary: Transformers have achieved great success in numerous NLP tasks but continue
  to exhibit notable gaps in multi-step factual reasoning, especially when
  real-world knowledge is sparse. Recent advances in grokking have demonstrated
  that neural networks can transition from memorizing to perfectly generalizing
  once they detect underlying logical patterns - yet these studies have primarily
  used small, synthetic tasks. In this paper, for the first time, we extend
  grokking to real-world factual data and address the challenge of dataset
  sparsity by augmenting existing knowledge graphs with carefully designed
  synthetic data to raise the ratio phi_r of inferred facts to atomic facts
  above the threshold required for grokking. Surprisingly, we find that even
  factually incorrect synthetic data can strengthen emergent reasoning circuits
  rather than degrade accuracy, as it forces the model to rely on relational
  structure rather than memorization. When evaluated on multi-hop reasoning
  benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA -
  substantially improving over strong baselines and matching or exceeding current
  state-of-the-art results. We further provide an in-depth analysis of how
  increasing phi_r drives the formation of generalizing circuits inside
  Transformers. Our findings suggest that grokking-based data augmentation can
  unlock implicit multi-hop reasoning capabilities, opening the door to more
  robust and interpretable factual reasoning in large-scale language models.
paper_authors._id: 6818c145daa8955b2085667d
paper_authors.name: Roman Abramov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T16:53:05.323Z
paper_authors._id: 63e7b92c4577a86987a53cd6
paper_authors.avatarUrl: /avatars/6cb0b5a1eaf4a84db663eeda96a3967d.svg
paper_authors.fullname: Roman Abramov
paper_authors.type: user
paper_authors.user: monsetrum
paper_authors._id: 6818c145daa8955b2085667e
paper_authors.name: Felix Steinbauer
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-06T13:01:28.304Z
paper_authors._id: 6679882913c63ebaa8ff62fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6679882913c63ebaa8ff62fe/zufYEHw7QNp50pfZx9SmF.jpeg
paper_authors.fullname: Felix Steinbauer
paper_authors.type: user
paper_authors.user: fsteinbauer
paper_authors._id: 6818c145daa8955b2085667f
paper_authors.name: Gjergji Kasneci
paper_title: Grokking in the Wild: Data Augmentation for Real-World Multi-Hop
    Reasoning with Transformers
paper_summary: Transformers have achieved great success in numerous NLP tasks but continue
  to exhibit notable gaps in multi-step factual reasoning, especially when
  real-world knowledge is sparse. Recent advances in grokking have demonstrated
  that neural networks can transition from memorizing to perfectly generalizing
  once they detect underlying logical patterns - yet these studies have primarily
  used small, synthetic tasks. In this paper, for the first time, we extend
  grokking to real-world factual data and address the challenge of dataset
  sparsity by augmenting existing knowledge graphs with carefully designed
  synthetic data to raise the ratio phi_r of inferred facts to atomic facts
  above the threshold required for grokking. Surprisingly, we find that even
  factually incorrect synthetic data can strengthen emergent reasoning circuits
  rather than degrade accuracy, as it forces the model to rely on relational
  structure rather than memorization. When evaluated on multi-hop reasoning
  benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA -
  substantially improving over strong baselines and matching or exceeding current
  state-of-the-art results. We further provide an in-depth analysis of how
  increasing phi_r drives the formation of generalizing circuits inside
  Transformers. Our findings suggest that grokking-based data augmentation can
  unlock implicit multi-hop reasoning capabilities, opening the door to more
  robust and interpretable factual reasoning in large-scale language models.
paper_ai_summary: Grokking with synthetic data augmentation in knowledge graphs enhances Transformer models' multi-step factual reasoning on real-world tasks, achieving high accuracy on benchmarks.
paper_ai_keywords: Transformers NLP factual reasoning grokking knowledge graphs synthetic data multi-hop reasoning 2WikiMultiHopQA generalizing circuits reasoning capabilities