Toward Grounded Social Reasoning
Consider a robot tasked with tidying a desk with a meticulously constructed
Lego sports car. A human may recognize that it is not socially appropriate to
disassemble the sports car and put it away as part of the "tidying". How can a
robot reach that conclusion? Although large language models (LLMs) have
recently been used to enable social reasoning, grounding this reasoning in the
real world has been challenging. To reason in the real world, robots must go
beyond passively querying LLMs and *actively gather information from the
environment* that is required to make the right decision. For instance, after
detecting that there is an occluded car, the robot may need to actively
perceive the car to know whether it is an advanced model car made out of Legos
or a toy car built by a toddler. We propose an approach that leverages an LLM
and vision language model (VLM) to help a robot actively perceive its
environment to perform grounded social reasoning. To evaluate our framework at
scale, we release the MessySurfaces dataset which contains images of 70
real-world surfaces that need to be cleaned. We additionally illustrate our
approach with a robot on 2 carefully designed surfaces. We find an average
12.9% improvement on the MessySurfaces benchmark and an average 15% improvement
on the robot experiments over baselines that do not use active perception. The
dataset, code, and videos of our approach can be found at
https://minaek.github.io/groundedsocialreasoning.
648bcf5ad60140c4bbccdcc9
Minae Kwon
admin_assigned
2023-06-19T11:48:18.565Z
645dd1f6a03f3ebb0be3d4de
/avatars/9a3ec029036403c3c57428c98fa1501c.svg
Minae Kwon
user
minaek
648bcf5ad60140c4bbccdcca
Hengyuan Hu
648bcf5ad60140c4bbccdccb
Vivek Myers
648bcf5ad60140c4bbccdccc
Siddharth Karamcheti
admin_assigned
2023-06-19T11:48:50.569Z
6150b090d84cf0532aa1764b
https://cdn-avatars.huggingface.co/v1/production/uploads/1634666324094-6150b090d84cf0532aa1764b.jpeg
Siddharth Karamcheti
user
skaramcheti
648bcf5ad60140c4bbccdccd
Anca Dragan
648bcf5ad60140c4bbccdcce
Dorsa Sadigh
Toward Grounded Social Reasoning
Consider a robot tasked with tidying a desk with a meticulously constructed
Lego sports car. A human may recognize that it is not socially appropriate to
disassemble the sports car and put it away as part of the "tidying". How can a
robot reach that conclusion? Although large language models (LLMs) have
recently been used to enable social reasoning, grounding this reasoning in the
real world has been challenging. To reason in the real world, robots must go
beyond passively querying LLMs and *actively gather information from the
environment* that is required to make the right decision. For instance, after
detecting that there is an occluded car, the robot may need to actively
perceive the car to know whether it is an advanced model car made out of Legos
or a toy car built by a toddler. We propose an approach that leverages an LLM
and vision language model (VLM) to help a robot actively perceive its
environment to perform grounded social reasoning. To evaluate our framework at
scale, we release the MessySurfaces dataset which contains images of 70
real-world surfaces that need to be cleaned. We additionally illustrate our
approach with a robot on 2 carefully designed surfaces. We find an average
12.9% improvement on the MessySurfaces benchmark and an average 15% improvement
on the robot experiments over baselines that do not use active perception. The
dataset, code, and videos of our approach can be found at
https://minaek.github.io/groundedsocialreasoning.
A robot uses an LLM and VLM to actively perceive its environment for grounded social reasoning, achieving improvements over baselines on a dataset of messy surfaces.
Large language models VLM active perception grounded social reasoning MessySurfaces dataset