Transformers meet Neural Algorithmic Reasoners
Transformers have revolutionized machine learning with their simple yet
effective architecture. Pre-training Transformers on massive text datasets from
the Internet has led to unmatched generalization for natural language
understanding (NLU) tasks. However, such language models remain fragile when
tasked with algorithmic forms of reasoning, where computations must be precise
and robust. To address this limitation, we propose a novel approach that
combines the Transformer's language understanding with the robustness of graph
neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs
proved effective as generic solvers for algorithmic tasks, when specified in
graph form. To make their embeddings accessible to a Transformer, we propose a
hybrid architecture with a two-phase training procedure, allowing the tokens in
the language model to cross-attend to the node embeddings from the NAR. We
evaluate our resulting TransNAR model on CLRS-Text, the text-based version of
the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only
models for algorithmic reasoning, both in and out of distribution.
666ba87cfa1b45be6fea76e6
Wilfried Bounsi
claimed_verified
2024-06-15T21:29:08.694Z
666d73d12c1f3a8ddfee14f0
/avatars/6bb75c10d1f0f67006781ccd03624500.svg
Wilfried Bounsi
user
wilcoln
666ba87cfa1b45be6fea76e7
Borja Ibarz
666ba87cfa1b45be6fea76e8
Andrew Dudzik
666ba87cfa1b45be6fea76e9
Jessica B. Hamrick
666ba87cfa1b45be6fea76ea
Larisa Markeeva
666ba87cfa1b45be6fea76eb
Alex Vitvitskyi
666ba87cfa1b45be6fea76ec
Razvan Pascanu
666ba87cfa1b45be6fea76ed
Petar Veličković
Transformers meet Neural Algorithmic Reasoners
Transformers have revolutionized machine learning with their simple yet
effective architecture. Pre-training Transformers on massive text datasets from
the Internet has led to unmatched generalization for natural language
understanding (NLU) tasks. However, such language models remain fragile when
tasked with algorithmic forms of reasoning, where computations must be precise
and robust. To address this limitation, we propose a novel approach that
combines the Transformer's language understanding with the robustness of graph
neural network (GNN)-based neural algorithmic reasoners (NARs). Such NARs
proved effective as generic solvers for algorithmic tasks, when specified in
graph form. To make their embeddings accessible to a Transformer, we propose a
hybrid architecture with a two-phase training procedure, allowing the tokens in
the language model to cross-attend to the node embeddings from the NAR. We
evaluate our resulting TransNAR model on CLRS-Text, the text-based version of
the CLRS-30 benchmark, and demonstrate significant gains over Transformer-only
models for algorithmic reasoning, both in and out of distribution.
A novel TransNAR model combines Transformer-based language understanding with graph neural network solvers to enhance algorithmic reasoning.
Transformers natural language understanding graph neural network neural algorithmic reasoners embeddings cross-attention TransNAR CLRS-Text CLRS-30 benchmark algorithmic reasoning