title: A Data Source for Reasoning Embodied Agents
summary: Recent progress in using machine learning models for reasoning tasks has been
  driven by novel model architectures, large-scale pre-training protocols, and
  dedicated reasoning datasets for fine-tuning. In this work, to further pursue
  these advances, we introduce a new data generator for machine reasoning that
  integrates with an embodied agent. The generated data consists of templated
  text queries and answers, matched with world-states encoded into a database.
  The world-states are a result of both world dynamics and the actions of the
  agent. We show the results of several baseline models on instantiations of
  train sets. These include pre-trained language models fine-tuned on a
  text-formatted representation of the database, and graph-structured
  Transformers operating on a knowledge-graph representation of the database. We
  find that these models can answer some questions about the world-state, but
  struggle with others. These results hint at new research directions in
  designing neural reasoning models and database representations. Code to
  generate the data will be released at github.com/facebookresearch/neuralmemory
paper_authors._id: 6507a684d5578ef7e2ff9514
paper_authors.name: Jack Lanchantin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:10:41.957Z
paper_authors._id: 65dd30a6ea1a202d3cb469f7
paper_authors.avatarUrl: /avatars/14029def026380e6e20822d916b25a72.svg
paper_authors.fullname: Jack Lanchantin
paper_authors.type: user
paper_authors.user: jcklcn
paper_authors._id: 6507a684d5578ef7e2ff9515
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-30T16:01:35.863Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 6507a684d5578ef7e2ff9516
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:33:39.054Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 6507a684d5578ef7e2ff9517
paper_authors.name: Yuxuan Sun
paper_authors._id: 6507a684d5578ef7e2ff9518
paper_authors.name: Kavya Srinet
paper_authors._id: 6507a684d5578ef7e2ff9519
paper_authors.name: Arthur Szlam
paper_title: A Data Source for Reasoning Embodied Agents
paper_summary: Recent progress in using machine learning models for reasoning tasks has been
  driven by novel model architectures, large-scale pre-training protocols, and
  dedicated reasoning datasets for fine-tuning. In this work, to further pursue
  these advances, we introduce a new data generator for machine reasoning that
  integrates with an embodied agent. The generated data consists of templated
  text queries and answers, matched with world-states encoded into a database.
  The world-states are a result of both world dynamics and the actions of the
  agent. We show the results of several baseline models on instantiations of
  train sets. These include pre-trained language models fine-tuned on a
  text-formatted representation of the database, and graph-structured
  Transformers operating on a knowledge-graph representation of the database. We
  find that these models can answer some questions about the world-state, but
  struggle with others. These results hint at new research directions in
  designing neural reasoning models and database representations. Code to
  generate the data will be released at github.com/facebookresearch/neuralmemory
paper_ai_keywords: data generator embodied agent templated text queries world-states database pre-trained language models graph-structured transformers knowledge-graph representation neural reasoning models database representations