title: InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
summary: We introduce InteractVLM, a novel method to estimate 3D contact points on
  human bodies and objects from single in-the-wild images, enabling accurate
  human-object joint reconstruction in 3D. This is challenging due to occlusions,
  depth ambiguities, and widely varying object shapes. Existing methods rely on
  3D contact annotations collected via expensive motion-capture systems or
  tedious manual labeling, limiting scalability and generalization. To overcome
  this, InteractVLM harnesses the broad visual knowledge of large Vision-Language
  Models (VLMs), fine-tuned with limited 3D contact data. However, directly
  applying these models is non-trivial, as they reason only in 2D, while
  human-object contact is inherently 3D. Thus we introduce a novel
  Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D
  space via multi-view rendering, (2) trains a novel multi-view localization
  model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D.
  Additionally, we propose a new task called Semantic Human Contact estimation,
  where human contact predictions are conditioned explicitly on object semantics,
  enabling richer interaction modeling. InteractVLM outperforms existing work on
  contact estimation and also facilitates 3D reconstruction from an in-the wild
  image. Code and models are available at https://interactvlm.is.tue.mpg.de.
paper_authors._id: 67fcbbe8daf0cf6803943949
paper_authors.name: Sai Kumar Dwivedi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-14T09:46:12.532Z
paper_authors._id: 6492bf9681d93008eb33f167
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6492bf9681d93008eb33f167/nMgmq7BXXPsF1y83JUeY3.jpeg
paper_authors.fullname: Sai Kumar Dwivedi
paper_authors.type: user
paper_authors.user: saidwivedi
paper_authors._id: 67fcbbe8daf0cf680394394a
paper_authors.name: Dimitrije Antić
paper_authors._id: 67fcbbe8daf0cf680394394b
paper_authors.name: Shashank Tripathi
paper_authors._id: 67fcbbe8daf0cf680394394c
paper_authors.name: Omid Taheri
paper_authors._id: 67fcbbe8daf0cf680394394d
paper_authors.name: Cordelia Schmid
paper_authors._id: 67fcbbe8daf0cf680394394e
paper_authors.name: Michael J. Black
paper_authors._id: 67fcbbe8daf0cf680394394f
paper_authors.name: Dimitrios Tzionas
paper_title: InteractVLM: 3D Interaction Reasoning from 2D Foundational Models
paper_summary: We introduce InteractVLM, a novel method to estimate 3D contact points on
  human bodies and objects from single in-the-wild images, enabling accurate
  human-object joint reconstruction in 3D. This is challenging due to occlusions,
  depth ambiguities, and widely varying object shapes. Existing methods rely on
  3D contact annotations collected via expensive motion-capture systems or
  tedious manual labeling, limiting scalability and generalization. To overcome
  this, InteractVLM harnesses the broad visual knowledge of large Vision-Language
  Models (VLMs), fine-tuned with limited 3D contact data. However, directly
  applying these models is non-trivial, as they reason only in 2D, while
  human-object contact is inherently 3D. Thus we introduce a novel
  Render-Localize-Lift module that: (1) embeds 3D body and object surfaces in 2D
  space via multi-view rendering, (2) trains a novel multi-view localization
  model (MV-Loc) to infer contacts in 2D, and (3) lifts these to 3D.
  Additionally, we propose a new task called Semantic Human Contact estimation,
  where human contact predictions are conditioned explicitly on object semantics,
  enabling richer interaction modeling. InteractVLM outperforms existing work on
  contact estimation and also facilitates 3D reconstruction from an in-the wild
  image. Code and models are available at https://interactvlm.is.tue.mpg.de.
paper_ai_summary: InteractVLM estimates 3D contact points using fine-tuned Vision-Language Models and a Render-Localize-Lift module, enhancing 3D human-object joint reconstruction with semantic contact predictions.
paper_ai_keywords: Vision-Language Models Render-Localize-Lift module multi-view rendering multi-view localization model Semantic Human Contact estimation