title: DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with
    Diffusion Priors
summary: We present DreamHOI, a novel method for zero-shot synthesis of human-object
  interactions (HOIs), enabling a 3D human model to realistically interact with
  any given object based on a textual description. This task is complicated by
  the varying categories and geometries of real-world objects and the scarcity of
  datasets encompassing diverse HOIs. To circumvent the need for extensive data,
  we leverage text-to-image diffusion models trained on billions of image-caption
  pairs. We optimize the articulation of a skinned human mesh using Score
  Distillation Sampling (SDS) gradients obtained from these models, which predict
  image-space edits. However, directly backpropagating image-space gradients into
  complex articulation parameters is ineffective due to the local nature of such
  gradients. To overcome this, we introduce a dual implicit-explicit
  representation of a skinned mesh, combining (implicit) neural radiance fields
  (NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization,
  we transition between implicit and explicit forms, grounding the NeRF
  generation while refining the mesh articulation. We validate our approach
  through extensive experiments, demonstrating its effectiveness in generating
  realistic HOIs.
paper_authors._id: 66e3ad9df910b65e13aeab91
paper_authors.name: Thomas Hanwen Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:59:40.126Z
paper_authors._id: 654ac8b57ed5797e22eb85e2
paper_authors.avatarUrl: /avatars/cb618f9e69ca0362a35b2f1bf4050064.svg
paper_authors.fullname: Thomas Zhu
paper_authors.type: user
paper_authors.user: hanwenzhu
paper_authors._id: 66e3ad9df910b65e13aeab92
paper_authors.name: Ruining Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T07:20:45.630Z
paper_authors._id: 6467cc17e7a6a374fd1a41e5
paper_authors.avatarUrl: /avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
paper_authors.fullname: Ruining Li
paper_authors.type: user
paper_authors.user: rayli
paper_authors._id: 66e3ad9df910b65e13aeab93
paper_authors.name: Tomas Jakab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-13T07:59:46.718Z
paper_authors._id: 6435823f76dbfd731bca13ab
paper_authors.avatarUrl: /avatars/67cd864bb8b1310a344e60a27b1ed075.svg
paper_authors.fullname: Tomas Jakab
paper_authors.type: user
paper_authors.user: tmsj
paper_title: DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with
    Diffusion Priors
paper_summary: We present DreamHOI, a novel method for zero-shot synthesis of human-object
  interactions (HOIs), enabling a 3D human model to realistically interact with
  any given object based on a textual description. This task is complicated by
  the varying categories and geometries of real-world objects and the scarcity of
  datasets encompassing diverse HOIs. To circumvent the need for extensive data,
  we leverage text-to-image diffusion models trained on billions of image-caption
  pairs. We optimize the articulation of a skinned human mesh using Score
  Distillation Sampling (SDS) gradients obtained from these models, which predict
  image-space edits. However, directly backpropagating image-space gradients into
  complex articulation parameters is ineffective due to the local nature of such
  gradients. To overcome this, we introduce a dual implicit-explicit
  representation of a skinned mesh, combining (implicit) neural radiance fields
  (NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization,
  we transition between implicit and explicit forms, grounding the NeRF
  generation while refining the mesh articulation. We validate our approach
  through extensive experiments, demonstrating its effectiveness in generating
  realistic HOIs.
paper_ai_summary: DreamHOI uses text-to-image diffusion models and a dual implicit-explicit skinned mesh representation to synthesize realistic zero-shot human-object interactions.
paper_ai_keywords: text-to-image diffusion models Score Distillation Sampling (SDS) skinned mesh neural radiance fields (NeRFs) skeleton-driven mesh articulation