title: Toward Grounded Social Reasoning
summary: Consider a robot tasked with tidying a desk with a meticulously constructed
  Lego sports car. A human may recognize that it is not socially appropriate to
  disassemble the sports car and put it away as part of the "tidying". How can a
  robot reach that conclusion? Although large language models (LLMs) have
  recently been used to enable social reasoning, grounding this reasoning in the
  real world has been challenging. To reason in the real world, robots must go
  beyond passively querying LLMs and *actively gather information from the
  environment* that is required to make the right decision. For instance, after
  detecting that there is an occluded car, the robot may need to actively
  perceive the car to know whether it is an advanced model car made out of Legos
  or a toy car built by a toddler. We propose an approach that leverages an LLM
  and vision language model (VLM) to help a robot actively perceive its
  environment to perform grounded social reasoning. To evaluate our framework at
  scale, we release the MessySurfaces dataset which contains images of 70
  real-world surfaces that need to be cleaned. We additionally illustrate our
  approach with a robot on 2 carefully designed surfaces. We find an average
  12.9% improvement on the MessySurfaces benchmark and an average 15% improvement
  on the robot experiments over baselines that do not use active perception. The
  dataset, code, and videos of our approach can be found at
  https://minaek.github.io/groundedsocialreasoning.
paper_authors._id: 648bcf5ad60140c4bbccdcc9
paper_authors.name: Minae Kwon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:48:18.565Z
paper_authors._id: 645dd1f6a03f3ebb0be3d4de
paper_authors.avatarUrl: /avatars/9a3ec029036403c3c57428c98fa1501c.svg
paper_authors.fullname: Minae Kwon
paper_authors.type: user
paper_authors.user: minaek
paper_authors._id: 648bcf5ad60140c4bbccdcca
paper_authors.name: Hengyuan Hu
paper_authors._id: 648bcf5ad60140c4bbccdccb
paper_authors.name: Vivek Myers
paper_authors._id: 648bcf5ad60140c4bbccdccc
paper_authors.name: Siddharth Karamcheti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:48:50.569Z
paper_authors._id: 6150b090d84cf0532aa1764b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634666324094-6150b090d84cf0532aa1764b.jpeg
paper_authors.fullname: Siddharth Karamcheti
paper_authors.type: user
paper_authors.user: skaramcheti
paper_authors._id: 648bcf5ad60140c4bbccdccd
paper_authors.name: Anca Dragan
paper_authors._id: 648bcf5ad60140c4bbccdcce
paper_authors.name: Dorsa Sadigh
paper_title: Toward Grounded Social Reasoning
paper_summary: Consider a robot tasked with tidying a desk with a meticulously constructed
  Lego sports car. A human may recognize that it is not socially appropriate to
  disassemble the sports car and put it away as part of the "tidying". How can a
  robot reach that conclusion? Although large language models (LLMs) have
  recently been used to enable social reasoning, grounding this reasoning in the
  real world has been challenging. To reason in the real world, robots must go
  beyond passively querying LLMs and *actively gather information from the
  environment* that is required to make the right decision. For instance, after
  detecting that there is an occluded car, the robot may need to actively
  perceive the car to know whether it is an advanced model car made out of Legos
  or a toy car built by a toddler. We propose an approach that leverages an LLM
  and vision language model (VLM) to help a robot actively perceive its
  environment to perform grounded social reasoning. To evaluate our framework at
  scale, we release the MessySurfaces dataset which contains images of 70
  real-world surfaces that need to be cleaned. We additionally illustrate our
  approach with a robot on 2 carefully designed surfaces. We find an average
  12.9% improvement on the MessySurfaces benchmark and an average 15% improvement
  on the robot experiments over baselines that do not use active perception. The
  dataset, code, and videos of our approach can be found at
  https://minaek.github.io/groundedsocialreasoning.
paper_ai_keywords: LLMs social reasoning grounding real world active gather information occluded advanced model car toy car toddler vision language model MessySurfaces dataset grounded social reasoning