DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with
  Diffusion Priors
We present DreamHOI, a novel method for zero-shot synthesis of human-object
interactions (HOIs), enabling a 3D human model to realistically interact with
any given object based on a textual description. This task is complicated by
the varying categories and geometries of real-world objects and the scarcity of
datasets encompassing diverse HOIs. To circumvent the need for extensive data,
we leverage text-to-image diffusion models trained on billions of image-caption
pairs. We optimize the articulation of a skinned human mesh using Score
Distillation Sampling (SDS) gradients obtained from these models, which predict
image-space edits. However, directly backpropagating image-space gradients into
complex articulation parameters is ineffective due to the local nature of such
gradients. To overcome this, we introduce a dual implicit-explicit
representation of a skinned mesh, combining (implicit) neural radiance fields
(NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization,
we transition between implicit and explicit forms, grounding the NeRF
generation while refining the mesh articulation. We validate our approach
through extensive experiments, demonstrating its effectiveness in generating
realistic HOIs.
66e3ad9df910b65e13aeab91
Thomas Hanwen Zhu
admin_assigned
2024-09-13T07:59:40.126Z
654ac8b57ed5797e22eb85e2
/avatars/cb618f9e69ca0362a35b2f1bf4050064.svg
Thomas Zhu
user
hanwenzhu
66e3ad9df910b65e13aeab92
Ruining Li
claimed_verified
2024-09-13T07:20:45.630Z
6467cc17e7a6a374fd1a41e5
/avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
Ruining Li
user
rayli
66e3ad9df910b65e13aeab93
Tomas Jakab
admin_assigned
2024-09-13T07:59:46.718Z
6435823f76dbfd731bca13ab
/avatars/67cd864bb8b1310a344e60a27b1ed075.svg
Tomas Jakab
user
tmsj
DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with
  Diffusion Priors
We present DreamHOI, a novel method for zero-shot synthesis of human-object
interactions (HOIs), enabling a 3D human model to realistically interact with
any given object based on a textual description. This task is complicated by
the varying categories and geometries of real-world objects and the scarcity of
datasets encompassing diverse HOIs. To circumvent the need for extensive data,
we leverage text-to-image diffusion models trained on billions of image-caption
pairs. We optimize the articulation of a skinned human mesh using Score
Distillation Sampling (SDS) gradients obtained from these models, which predict
image-space edits. However, directly backpropagating image-space gradients into
complex articulation parameters is ineffective due to the local nature of such
gradients. To overcome this, we introduce a dual implicit-explicit
representation of a skinned mesh, combining (implicit) neural radiance fields
(NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization,
we transition between implicit and explicit forms, grounding the NeRF
generation while refining the mesh articulation. We validate our approach
through extensive experiments, demonstrating its effectiveness in generating
realistic HOIs.
DreamHOI uses text-to-image diffusion models and a dual implicit-explicit skinned mesh representation to synthesize realistic zero-shot human-object interactions.
text-to-image diffusion models Score Distillation Sampling (SDS) skinned mesh neural radiance fields (NeRFs) skeleton-driven mesh articulation