EgoNormia: Benchmarking Physical Social Norm Understanding
Human activity is moderated by norms. When performing actions in the real
world, humans not only follow norms, but also consider the trade-off between
different norms However, machines are often trained without explicit
supervision on norm understanding and reasoning, especially when the norms are
grounded in a physical and social context. To improve and evaluate the
normative reasoning capability of vision-language models (VLMs), we present
EgoNormia |epsilon|, consisting of 1,853 ego-centric videos of human
interactions, each of which has two related questions evaluating both the
prediction and justification of normative actions. The normative actions
encompass seven categories: safety, privacy, proxemics, politeness,
cooperation, coordination/proactivity, and communication/legibility. To compile
this dataset at scale, we propose a novel pipeline leveraging video sampling,
automatic answer generation, filtering, and human validation. Our work
demonstrates that current state-of-the-art vision-language models lack robust
norm understanding, scoring a maximum of 45% on EgoNormia (versus a human bench
of 92%). Our analysis of performance in each dimension highlights the
significant risks of safety, privacy, and the lack of collaboration and
communication capability when applied to real-world agents. We additionally
show that through a retrieval-based generation method, it is possible to use
EgoNomia to enhance normative reasoning in VLMs.
67c5c853e7c5cfb1d2b52858
MohammadHossein Rezaei
extracted_confirmed
2025-03-03T16:56:51.354Z
63f6ba02a67b8acfa50407bb
https://cdn-avatars.huggingface.co/v1/production/uploads/63f6ba02a67b8acfa50407bb/ueUb01p1mhuRNrkyfEHtc.jpeg
MohammadHossein Rezaei
user
mhr2004
67c5c853e7c5cfb1d2b52859
Yicheng Fu
67c5c853e7c5cfb1d2b5285a
Phil Cuvin
67c5c853e7c5cfb1d2b5285b
Caleb Ziems
67c5c853e7c5cfb1d2b5285c
Yanzhe Zhang
67c5c853e7c5cfb1d2b5285d
Hao Zhu
claimed_verified
2025-03-03T16:07:08.219Z
61aa376688c20eebf1e8deb3
/avatars/7c11dcb232c73547d7d87834be287822.svg
Hao Zhu
user
ProKil
67c5c853e7c5cfb1d2b5285e
Diyi Yang
EgoNormia: Benchmarking Physical Social Norm Understanding
Human activity is moderated by norms. When performing actions in the real
world, humans not only follow norms, but also consider the trade-off between
different norms However, machines are often trained without explicit
supervision on norm understanding and reasoning, especially when the norms are
grounded in a physical and social context. To improve and evaluate the
normative reasoning capability of vision-language models (VLMs), we present
EgoNormia |epsilon|, consisting of 1,853 ego-centric videos of human
interactions, each of which has two related questions evaluating both the
prediction and justification of normative actions. The normative actions
encompass seven categories: safety, privacy, proxemics, politeness,
cooperation, coordination/proactivity, and communication/legibility. To compile
this dataset at scale, we propose a novel pipeline leveraging video sampling,
automatic answer generation, filtering, and human validation. Our work
demonstrates that current state-of-the-art vision-language models lack robust
norm understanding, scoring a maximum of 45% on EgoNormia (versus a human bench
of 92%). Our analysis of performance in each dimension highlights the
significant risks of safety, privacy, and the lack of collaboration and
communication capability when applied to real-world agents. We additionally
show that through a retrieval-based generation method, it is possible to use
EgoNomia to enhance normative reasoning in VLMs.
EgoNormia, a dataset of ego-centric videos with normative action questions, demonstrates limitations in state-of-the-art vision-language models' understanding of social norms and highlights the need for improvements in safety, privacy, and collaboration.
ego-centric videos vision-language models normative reasoning video sampling automatic answer generation filtering human validation normative actions safety privacy proxemics politeness cooperation coordination/proactivity communication/legibility retrieval-based generation