NIFTY: Neural Object Interaction Fields for Guided Human Motion
  Synthesis
We address the problem of generating realistic 3D motions of humans
interacting with objects in a scene. Our key idea is to create a neural
interaction field attached to a specific object, which outputs the distance to
the valid interaction manifold given a human pose as input. This interaction
field guides the sampling of an object-conditioned human motion diffusion
model, so as to encourage plausible contacts and affordance semantics. To
support interactions with scarcely available data, we propose an automated
synthetic data pipeline. For this, we seed a pre-trained motion model, which
has priors for the basics of human movement, with interaction-specific anchor
poses extracted from limited motion capture data. Using our guided diffusion
model trained on generated synthetic data, we synthesize realistic motions for
sitting and lifting with several objects, outperforming alternative approaches
in terms of motion quality and successful action completion. We call our
framework NIFTY: Neural Interaction Fields for Trajectory sYnthesis.
64b4ada9520fa3a154ff746d
Nilesh Kulkarni
admin_assigned
2023-07-17T11:10:42.289Z
63a8e340353e10031a90cf40
/avatars/eee97deed32e1ba366e2216dd159988b.svg
Nilesh Kulkarni
user
nileshk
64b4ada9520fa3a154ff746e
Davis Rempe
admin_assigned
2023-07-17T11:10:56.980Z
62f9e6790697d2242198e08d
/avatars/693792462687f5e2485bec9cd6bee16f.svg
Davis Rempe
user
drempe
64b4ada9520fa3a154ff746f
Kyle Genova
64b4ada9520fa3a154ff7470
Abhijit Kundu
64b4ada9520fa3a154ff7471
Justin Johnson
64b4ada9520fa3a154ff7472
David Fouhey
64b4ada9520fa3a154ff7473
Leonidas Guibas
NIFTY: Neural Object Interaction Fields for Guided Human Motion
  Synthesis
We address the problem of generating realistic 3D motions of humans
interacting with objects in a scene. Our key idea is to create a neural
interaction field attached to a specific object, which outputs the distance to
the valid interaction manifold given a human pose as input. This interaction
field guides the sampling of an object-conditioned human motion diffusion
model, so as to encourage plausible contacts and affordance semantics. To
support interactions with scarcely available data, we propose an automated
synthetic data pipeline. For this, we seed a pre-trained motion model, which
has priors for the basics of human movement, with interaction-specific anchor
poses extracted from limited motion capture data. Using our guided diffusion
model trained on generated synthetic data, we synthesize realistic motions for
sitting and lifting with several objects, outperforming alternative approaches
in terms of motion quality and successful action completion. We call our
framework NIFTY: Neural Interaction Fields for Trajectory sYnthesis.
A framework uses neural interaction fields and diffusion models to generate realistic 3D human motions interacting with objects using limited data.
neural interaction field interaction manifold human pose object-conditioned human motion diffusion model plausible contacts affordance semantics synthetic data pipeline pre-trained motion model interaction-specific anchor poses guided diffusion model motion quality successful action completion