INTRA: Interaction Relationship-aware Weakly Supervised Affordance
  Grounding
Affordance denotes the potential interactions inherent in objects. The
perception of affordance can enable intelligent agents to navigate and interact
with new environments efficiently. Weakly supervised affordance grounding
teaches agents the concept of affordance without costly pixel-level
annotations, but with exocentric images. Although recent advances in weakly
supervised affordance grounding yielded promising results, there remain
challenges including the requirement for paired exocentric and egocentric image
dataset, and the complexity in grounding diverse affordances for a single
object. To address them, we propose INTeraction Relationship-aware weakly
supervised Affordance grounding (INTRA). Unlike prior arts, INTRA recasts this
problem as representation learning to identify unique features of interactions
through contrastive learning with exocentric images only, eliminating the need
for paired datasets. Moreover, we leverage vision-language model embeddings for
performing affordance grounding flexibly with any text, designing
text-conditioned affordance map generation to reflect interaction relationship
for contrastive learning and enhancing robustness with our text synonym
augmentation. Our method outperformed prior arts on diverse datasets such as
AGD20K, IIT-AFF, CAD and UMD. Additionally, experimental results demonstrate
that our method has remarkable domain scalability for synthesized images /
illustrations and is capable of performing affordance grounding for novel
interactions and objects.
66e11334a533df3d825ffe22
Ji Ha Jang
claimed_verified
2024-09-11T07:49:07.388Z
64c77dc26a26cddbec858bcb
/avatars/635e64f8205c194e63876cba9d8c5449.svg
jihajang
user
jeeit17
66e11334a533df3d825ffe23
Hoigi Seo
claimed_verified
2024-09-11T07:26:15.433Z
633e6f07309a99325095dd42
/avatars/57b91a488ac1745b3c0509c04eb6ad93.svg
Hoigi Seo
user
Agorium
66e11334a533df3d825ffe24
Se Young Chun
INTRA: Interaction Relationship-aware Weakly Supervised Affordance
  Grounding
Affordance denotes the potential interactions inherent in objects. The
perception of affordance can enable intelligent agents to navigate and interact
with new environments efficiently. Weakly supervised affordance grounding
teaches agents the concept of affordance without costly pixel-level
annotations, but with exocentric images. Although recent advances in weakly
supervised affordance grounding yielded promising results, there remain
challenges including the requirement for paired exocentric and egocentric image
dataset, and the complexity in grounding diverse affordances for a single
object. To address them, we propose INTeraction Relationship-aware weakly
supervised Affordance grounding (INTRA). Unlike prior arts, INTRA recasts this
problem as representation learning to identify unique features of interactions
through contrastive learning with exocentric images only, eliminating the need
for paired datasets. Moreover, we leverage vision-language model embeddings for
performing affordance grounding flexibly with any text, designing
text-conditioned affordance map generation to reflect interaction relationship
for contrastive learning and enhancing robustness with our text synonym
augmentation. Our method outperformed prior arts on diverse datasets such as
AGD20K, IIT-AFF, CAD and UMD. Additionally, experimental results demonstrate
that our method has remarkable domain scalability for synthesized images /
illustrations and is capable of performing affordance grounding for novel
interactions and objects.
INTRA addresses challenges in weakly supervised affordance grounding by using contrastive learning with exocentric images and vision-language embeddings, enhancing flexibility and robustness.
weakly supervised affordance grounding exocentric images egocentric image dataset representation learning contrastive learning vision-language model embeddings text-conditioned affordance map generation text synonym augmentation AGD20K IIT-AFF CAD UMD domain scalability synthesized images illustrations