title: Grounding Language in Multi-Perspective Referential Communication
summary: We introduce a task and dataset for referring expression generation and
  comprehension in multi-agent embodied environments. In this task, two agents in
  a shared scene must take into account one another's visual perspective, which
  may be different from their own, to both produce and understand references to
  objects in a scene and the spatial relations between them. We collect a dataset
  of 2,970 human-written referring expressions, each paired with human
  comprehension judgments, and evaluate the performance of automated models as
  speakers and listeners paired with human partners, finding that model
  performance in both reference generation and comprehension lags behind that of
  pairs of human agents. Finally, we experiment training an open-weight speaker
  model with evidence of communicative success when paired with a listener,
  resulting in an improvement from 58.9 to 69.3% in communicative success and
  even outperforming the strongest proprietary model.
paper_authors._id: 6704997d2a1ddf444eb8ef14
paper_authors.name: Zineng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:53:51.086Z
paper_authors._id: 6326e20bf0e99f96e024a164
paper_authors.avatarUrl: /avatars/9be111af274cad965f72ed48c71d6122.svg
paper_authors.fullname: Zineng Tang
paper_authors.type: user
paper_authors.user: ZinengTang
paper_authors._id: 6704997d2a1ddf444eb8ef15
paper_authors.name: Lingjun Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:53:56.765Z
paper_authors._id: 65509836e5f380aca800c408
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65509836e5f380aca800c408/wS6edLhx3YNaWWj0o8kaV.jpeg
paper_authors.fullname: Lingjun Mao
paper_authors.type: user
paper_authors.user: mao1207
paper_authors._id: 6704997d2a1ddf444eb8ef16
paper_authors.name: Alane Suhr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-08T09:54:04.425Z
paper_authors._id: 6611e6e1188ff298b0dd0b79
paper_authors.avatarUrl: /avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
paper_authors.fullname: Alane Suhr
paper_authors.type: user
paper_authors.user: alsuhr
paper_title: Grounding Language in Multi-Perspective Referential Communication
paper_summary: We introduce a task and dataset for referring expression generation and
  comprehension in multi-agent embodied environments. In this task, two agents in
  a shared scene must take into account one another's visual perspective, which
  may be different from their own, to both produce and understand references to
  objects in a scene and the spatial relations between them. We collect a dataset
  of 2,970 human-written referring expressions, each paired with human
  comprehension judgments, and evaluate the performance of automated models as
  speakers and listeners paired with human partners, finding that model
  performance in both reference generation and comprehension lags behind that of
  pairs of human agents. Finally, we experiment training an open-weight speaker
  model with evidence of communicative success when paired with a listener,
  resulting in an improvement from 58.9 to 69.3% in communicative success and
  even outperforming the strongest proprietary model.
paper_ai_summary: Agents in shared environments generate and comprehend referring expressions, showing human superiority initially, but a trained speaker model improves communication success when paired with a listener.
paper_ai_keywords: referring expression generation comprehension multi-agent embodied environments visual perspective spatial relations referring expressions human comprehension judgments automated models open-weight speaker model communicative success