Grokking in the Wild: Data Augmentation for Real-World Multi-Hop
  Reasoning with Transformers
Transformers have achieved great success in numerous NLP tasks but continue
to exhibit notable gaps in multi-step factual reasoning, especially when
real-world knowledge is sparse. Recent advances in grokking have demonstrated
that neural networks can transition from memorizing to perfectly generalizing
once they detect underlying logical patterns - yet these studies have primarily
used small, synthetic tasks. In this paper, for the first time, we extend
grokking to real-world factual data and address the challenge of dataset
sparsity by augmenting existing knowledge graphs with carefully designed
synthetic data to raise the ratio phi_r of inferred facts to atomic facts
above the threshold required for grokking. Surprisingly, we find that even
factually incorrect synthetic data can strengthen emergent reasoning circuits
rather than degrade accuracy, as it forces the model to rely on relational
structure rather than memorization. When evaluated on multi-hop reasoning
benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA -
substantially improving over strong baselines and matching or exceeding current
state-of-the-art results. We further provide an in-depth analysis of how
increasing phi_r drives the formation of generalizing circuits inside
Transformers. Our findings suggest that grokking-based data augmentation can
unlock implicit multi-hop reasoning capabilities, opening the door to more
robust and interpretable factual reasoning in large-scale language models.
6818c145daa8955b2085667d
Roman Abramov
claimed_verified
2025-05-06T16:53:05.323Z
63e7b92c4577a86987a53cd6
/avatars/6cb0b5a1eaf4a84db663eeda96a3967d.svg
Roman Abramov
user
monsetrum
6818c145daa8955b2085667e
Felix Steinbauer
extracted_confirmed
2025-05-06T13:01:28.304Z
6679882913c63ebaa8ff62fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6679882913c63ebaa8ff62fe/zufYEHw7QNp50pfZx9SmF.jpeg
Felix Steinbauer
user
fsteinbauer
6818c145daa8955b2085667f
Gjergji Kasneci
Grokking in the Wild: Data Augmentation for Real-World Multi-Hop
  Reasoning with Transformers
Transformers have achieved great success in numerous NLP tasks but continue
to exhibit notable gaps in multi-step factual reasoning, especially when
real-world knowledge is sparse. Recent advances in grokking have demonstrated
that neural networks can transition from memorizing to perfectly generalizing
once they detect underlying logical patterns - yet these studies have primarily
used small, synthetic tasks. In this paper, for the first time, we extend
grokking to real-world factual data and address the challenge of dataset
sparsity by augmenting existing knowledge graphs with carefully designed
synthetic data to raise the ratio phi_r of inferred facts to atomic facts
above the threshold required for grokking. Surprisingly, we find that even
factually incorrect synthetic data can strengthen emergent reasoning circuits
rather than degrade accuracy, as it forces the model to rely on relational
structure rather than memorization. When evaluated on multi-hop reasoning
benchmarks, our approach achieves up to 95-100% accuracy on 2WikiMultiHopQA -
substantially improving over strong baselines and matching or exceeding current
state-of-the-art results. We further provide an in-depth analysis of how
increasing phi_r drives the formation of generalizing circuits inside
Transformers. Our findings suggest that grokking-based data augmentation can
unlock implicit multi-hop reasoning capabilities, opening the door to more
robust and interpretable factual reasoning in large-scale language models.
Grokking with synthetic data augmentation in knowledge graphs enhances Transformer models' multi-step factual reasoning on real-world tasks, achieving high accuracy on benchmarks.
Transformers NLP factual reasoning grokking knowledge graphs synthetic data multi-hop reasoning 2WikiMultiHopQA generalizing circuits reasoning capabilities