title: EgoNormia: Benchmarking Physical Social Norm Understanding
summary: Human activity is moderated by norms. When performing actions in the real
  world, humans not only follow norms, but also consider the trade-off between
  different norms However, machines are often trained without explicit
  supervision on norm understanding and reasoning, especially when the norms are
  grounded in a physical and social context. To improve and evaluate the
  normative reasoning capability of vision-language models (VLMs), we present
  EgoNormia |epsilon|, consisting of 1,853 ego-centric videos of human
  interactions, each of which has two related questions evaluating both the
  prediction and justification of normative actions. The normative actions
  encompass seven categories: safety, privacy, proxemics, politeness,
  cooperation, coordination/proactivity, and communication/legibility. To compile
  this dataset at scale, we propose a novel pipeline leveraging video sampling,
  automatic answer generation, filtering, and human validation. Our work
  demonstrates that current state-of-the-art vision-language models lack robust
  norm understanding, scoring a maximum of 45% on EgoNormia (versus a human bench
  of 92%). Our analysis of performance in each dimension highlights the
  significant risks of safety, privacy, and the lack of collaboration and
  communication capability when applied to real-world agents. We additionally
  show that through a retrieval-based generation method, it is possible to use
  EgoNomia to enhance normative reasoning in VLMs.
paper_authors._id: 67c5c853e7c5cfb1d2b52858
paper_authors.name: MohammadHossein Rezaei
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-03T16:56:51.354Z
paper_authors._id: 63f6ba02a67b8acfa50407bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f6ba02a67b8acfa50407bb/ueUb01p1mhuRNrkyfEHtc.jpeg
paper_authors.fullname: MohammadHossein Rezaei
paper_authors.type: user
paper_authors.user: mhr2004
paper_authors._id: 67c5c853e7c5cfb1d2b52859
paper_authors.name: Yicheng Fu
paper_authors._id: 67c5c853e7c5cfb1d2b5285a
paper_authors.name: Phil Cuvin
paper_authors._id: 67c5c853e7c5cfb1d2b5285b
paper_authors.name: Caleb Ziems
paper_authors._id: 67c5c853e7c5cfb1d2b5285c
paper_authors.name: Yanzhe Zhang
paper_authors._id: 67c5c853e7c5cfb1d2b5285d
paper_authors.name: Hao Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-03T16:07:08.219Z
paper_authors._id: 61aa376688c20eebf1e8deb3
paper_authors.avatarUrl: /avatars/7c11dcb232c73547d7d87834be287822.svg
paper_authors.fullname: Hao Zhu
paper_authors.type: user
paper_authors.user: ProKil
paper_authors._id: 67c5c853e7c5cfb1d2b5285e
paper_authors.name: Diyi Yang
paper_title: EgoNormia: Benchmarking Physical Social Norm Understanding
paper_summary: Human activity is moderated by norms. When performing actions in the real
  world, humans not only follow norms, but also consider the trade-off between
  different norms However, machines are often trained without explicit
  supervision on norm understanding and reasoning, especially when the norms are
  grounded in a physical and social context. To improve and evaluate the
  normative reasoning capability of vision-language models (VLMs), we present
  EgoNormia |epsilon|, consisting of 1,853 ego-centric videos of human
  interactions, each of which has two related questions evaluating both the
  prediction and justification of normative actions. The normative actions
  encompass seven categories: safety, privacy, proxemics, politeness,
  cooperation, coordination/proactivity, and communication/legibility. To compile
  this dataset at scale, we propose a novel pipeline leveraging video sampling,
  automatic answer generation, filtering, and human validation. Our work
  demonstrates that current state-of-the-art vision-language models lack robust
  norm understanding, scoring a maximum of 45% on EgoNormia (versus a human bench
  of 92%). Our analysis of performance in each dimension highlights the
  significant risks of safety, privacy, and the lack of collaboration and
  communication capability when applied to real-world agents. We additionally
  show that through a retrieval-based generation method, it is possible to use
  EgoNomia to enhance normative reasoning in VLMs.