InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
We introduce InteractVLM, a novel method to estimate 3D contact points on
human bodies and objects from single in-the-wild images, enabling accurate
human-object joint reconstruction in 3D. This is challenging due to occlusions,
depth ambiguities, and widely varying object shapes. Existing methods rely on
3D contact annotations collected via expensive motion-capture systems or
tedious manual labeling, limiting scalability and generalization. To overcome
this, InteractVLM harnesses the broad visual knowledge of large Vision-Language
Models (VLMs), fine-tuned with limited 3D contact data. However, directly
applying these models is non-trivial, as they reason only in 2D, while
human-object contact is inherently 3D. Thus we introduce a novel
Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D
space via multi-view rendering, (2) trains a novel multi-view localization
model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D.
Additionally, we propose a new task called Semantic Human Contact estimation,
where human contact predictions are conditioned explicitly on object semantics,
enabling richer interaction modeling. InteractVLM outperforms existing work on
contact estimation and also facilitates 3D reconstruction from an in-the wild
image. Code and models are available at https://interactvlm.is.tue.mpg.de.
67fcbbe8daf0cf6803943949
Sai Kumar Dwivedi
claimed_verified
2025-04-14T09:46:12.532Z
6492bf9681d93008eb33f167
https://cdn-avatars.huggingface.co/v1/production/uploads/6492bf9681d93008eb33f167/nMgmq7BXXPsF1y83JUeY3.jpeg
Sai Kumar Dwivedi
user
saidwivedi
67fcbbe8daf0cf680394394a
Dimitrije Antić
67fcbbe8daf0cf680394394b
Shashank Tripathi
67fcbbe8daf0cf680394394c
Omid Taheri
67fcbbe8daf0cf680394394d
Cordelia Schmid
67fcbbe8daf0cf680394394e
Michael J. Black
67fcbbe8daf0cf680394394f
Dimitrios Tzionas
InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
We introduce InteractVLM, a novel method to estimate 3D contact points on
human bodies and objects from single in-the-wild images, enabling accurate
human-object joint reconstruction in 3D. This is challenging due to occlusions,
depth ambiguities, and widely varying object shapes. Existing methods rely on
3D contact annotations collected via expensive motion-capture systems or
tedious manual labeling, limiting scalability and generalization. To overcome
this, InteractVLM harnesses the broad visual knowledge of large Vision-Language
Models (VLMs), fine-tuned with limited 3D contact data. However, directly
applying these models is non-trivial, as they reason only in 2D, while
human-object contact is inherently 3D. Thus we introduce a novel
Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D
space via multi-view rendering, (2) trains a novel multi-view localization
model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D.
Additionally, we propose a new task called Semantic Human Contact estimation,
where human contact predictions are conditioned explicitly on object semantics,
enabling richer interaction modeling. InteractVLM outperforms existing work on
contact estimation and also facilitates 3D reconstruction from an in-the wild
image. Code and models are available at https://interactvlm.is.tue.mpg.de.
InteractVLM estimates 3D contact points using fine-tuned Vision-Language Models and a Render-Localize-Lift module, enhancing 3D human-object joint reconstruction with semantic contact predictions.
Vision-Language Models Render-Localize-Lift module multi-view rendering multi-view localization model Semantic Human Contact estimation