title: Transformers meet Neural Algorithmic Reasoners
summary: Transformers have revolutionized machine learning with their simple yet
  effective architecture. Pre-training Transformers on massive text datasets from
  the Internet has led to unmatched generalization for natural language
  understanding (NLU) tasks. However, such language models remain fragile when
  tasked with algorithmic forms of reasoning, where computations must be precise
  and robust. To address this limitation, we propose a novel approach that
  combines the Transformer's language understanding with the robustness of graph
  neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs
  proved effective as generic solvers for algorithmic tasks, when specified in
  graph form. To make their embeddings accessible to a Transformer, we propose a
  hybrid architecture with a two-phase training procedure, allowing the tokens in
  the language model to cross-attend to the node embeddings from the NAR. We
  evaluate our resulting TransNAR model on CLRS-Text, the text-based version of
  the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only
  models for algorithmic reasoning, both in and out of distribution.
paper_authors._id: 666ba87cfa1b45be6fea76e6
paper_authors.name: Wilfried Bounsi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-15T21:29:08.694Z
paper_authors._id: 666d73d12c1f3a8ddfee14f0
paper_authors.avatarUrl: /avatars/6bb75c10d1f0f67006781ccd03624500.svg
paper_authors.fullname: Wilfried Bounsi
paper_authors.type: user
paper_authors.user: wilcoln
paper_authors._id: 666ba87cfa1b45be6fea76e7
paper_authors.name: Borja Ibarz
paper_authors._id: 666ba87cfa1b45be6fea76e8
paper_authors.name: Andrew Dudzik
paper_authors._id: 666ba87cfa1b45be6fea76e9
paper_authors.name: Jessica B. Hamrick
paper_authors._id: 666ba87cfa1b45be6fea76ea
paper_authors.name: Larisa Markeeva
paper_authors._id: 666ba87cfa1b45be6fea76eb
paper_authors.name: Alex Vitvitskyi
paper_authors._id: 666ba87cfa1b45be6fea76ec
paper_authors.name: Razvan Pascanu
paper_authors._id: 666ba87cfa1b45be6fea76ed
paper_authors.name: Petar Veličković
paper_title: Transformers meet Neural Algorithmic Reasoners
paper_summary: Transformers have revolutionized machine learning with their simple yet
  effective architecture. Pre-training Transformers on massive text datasets from
  the Internet has led to unmatched generalization for natural language
  understanding (NLU) tasks. However, such language models remain fragile when
  tasked with algorithmic forms of reasoning, where computations must be precise
  and robust. To address this limitation, we propose a novel approach that
  combines the Transformer's language understanding with the robustness of graph
  neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs
  proved effective as generic solvers for algorithmic tasks, when specified in
  graph form. To make their embeddings accessible to a Transformer, we propose a
  hybrid architecture with a two-phase training procedure, allowing the tokens in
  the language model to cross-attend to the node embeddings from the NAR. We
  evaluate our resulting TransNAR model on CLRS-Text, the text-based version of
  the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only
  models for algorithmic reasoning, both in and out of distribution.
paper_ai_summary: A novel TransNAR model combines Transformer-based language understanding with graph neural network solvers to enhance algorithmic reasoning.
paper_ai_keywords: Transformers natural language understanding graph neural network neural algorithmic reasoners embeddings cross-attention TransNAR CLRS-Text CLRS-30 benchmark algorithmic reasoning