title: When can transformers reason with abstract symbols?
summary: We investigate the capabilities of transformer large language models (LLMs)
  on relational reasoning tasks involving abstract symbols. Such tasks have long
  been studied in the neuroscience literature as fundamental building blocks for
  more complex abilities in programming, mathematics, and verbal reasoning. For
  (i) regression tasks, we prove that transformers generalize when trained, but
  require astonishingly large quantities of training data. For (ii)
  next-token-prediction tasks with symbolic labels, we show an "inverse scaling
  law": transformers fail to generalize as their embedding dimension increases.
  For both settings (i) and (ii), we propose subtle transformer modifications
  which can reduce the amount of data needed by adding two trainable parameters
  per head.
paper_authors._id: 652decb82d617ec278bf9c5b
paper_authors.name: Enric Boix-Adsera
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-17T20:06:23.846Z
paper_authors._id: 646e2a732f1c836fa8450786
paper_authors.avatarUrl: /avatars/da8df192767bf2a2a16a3889118b5184.svg
paper_authors.fullname: Enric Boix
paper_authors.type: user
paper_authors.user: eboix
paper_authors._id: 652decb82d617ec278bf9c5c
paper_authors.name: Omid Saremi
paper_authors._id: 652decb82d617ec278bf9c5d
paper_authors.name: Emmanuel Abbe
paper_authors._id: 652decb82d617ec278bf9c5e
paper_authors.name: Samy Bengio
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:07:47.351Z
paper_authors._id: 6488d89678570cedad6f4e03
paper_authors.avatarUrl: /avatars/d3c86914cab46b953c0ee7ed17e3d6bb.svg
paper_authors.fullname: Samy Bengio
paper_authors.type: user
paper_authors.user: bengio
paper_authors._id: 652decb82d617ec278bf9c5f
paper_authors.name: Etai Littwin
paper_authors._id: 652decb82d617ec278bf9c60
paper_authors.name: Joshua Susskind
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-17T16:20:19.756Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_title: When can transformers reason with abstract symbols?
paper_summary: We investigate the capabilities of transformer large language models (LLMs)
  on relational reasoning tasks involving abstract symbols. Such tasks have long
  been studied in the neuroscience literature as fundamental building blocks for
  more complex abilities in programming, mathematics, and verbal reasoning. For
  (i) regression tasks, we prove that transformers generalize when trained, but
  require astonishingly large quantities of training data. For (ii)
  next-token-prediction tasks with symbolic labels, we show an "inverse scaling
  law": transformers fail to generalize as their embedding dimension increases.
  For both settings (i) and (ii), we propose subtle transformer modifications
  which can reduce the amount of data needed by adding two trainable parameters
  per head.
paper_ai_keywords: transformer large language models (LLMs) relational reasoning tasks abstract symbols regression tasks next-token-prediction tasks symbolic labels embedding dimension inverse scaling law