Grounding Language in Multi-Perspective Referential Communication
We introduce a task and dataset for referring expression generation and
comprehension in multi-agent embodied environments. In this task, two agents in
a shared scene must take into account one another's visual perspective, which
may be different from their own, to both produce and understand references to
objects in a scene and the spatial relations between them. We collect a dataset
of 2,970 human-written referring expressions, each paired with human
comprehension judgments, and evaluate the performance of automated models as
speakers and listeners paired with human partners, finding that model
performance in both reference generation and comprehension lags behind that of
pairs of human agents. Finally, we experiment training an open-weight speaker
model with evidence of communicative success when paired with a listener,
resulting in an improvement from 58.9 to 69.3% in communicative success and
even outperforming the strongest proprietary model.
6704997d2a1ddf444eb8ef14
Zineng Tang
admin_assigned
2024-10-08T09:53:51.086Z
6326e20bf0e99f96e024a164
/avatars/9be111af274cad965f72ed48c71d6122.svg
Zineng Tang
user
ZinengTang
6704997d2a1ddf444eb8ef15
Lingjun Mao
admin_assigned
2024-10-08T09:53:56.765Z
65509836e5f380aca800c408
https://cdn-avatars.huggingface.co/v1/production/uploads/65509836e5f380aca800c408/wS6edLhx3YNaWWj0o8kaV.jpeg
Lingjun Mao
user
mao1207
6704997d2a1ddf444eb8ef16
Alane Suhr
admin_assigned
2024-10-08T09:54:04.425Z
6611e6e1188ff298b0dd0b79
/avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
Alane Suhr
user
alsuhr
Grounding Language in Multi-Perspective Referential Communication
We introduce a task and dataset for referring expression generation and
comprehension in multi-agent embodied environments. In this task, two agents in
a shared scene must take into account one another's visual perspective, which
may be different from their own, to both produce and understand references to
objects in a scene and the spatial relations between them. We collect a dataset
of 2,970 human-written referring expressions, each paired with human
comprehension judgments, and evaluate the performance of automated models as
speakers and listeners paired with human partners, finding that model
performance in both reference generation and comprehension lags behind that of
pairs of human agents. Finally, we experiment training an open-weight speaker
model with evidence of communicative success when paired with a listener,
resulting in an improvement from 58.9 to 69.3% in communicative success and
even outperforming the strongest proprietary model.
Agents in shared environments generate and comprehend referring expressions, showing human superiority initially, but a trained speaker model improves communication success when paired with a listener.
referring expression generation comprehension multi-agent embodied environments visual perspective spatial relations referring expressions human comprehension judgments automated models open-weight speaker model communicative success