MACS: Mass Conditioned 3D Hand and Object Motion Synthesis
The physical properties of an object, such as mass, significantly affect how
we manipulate it with our hands. Surprisingly, this aspect has so far been
neglected in prior work on 3D motion synthesis. To improve the naturalness of
the synthesized 3D hand object motions, this work proposes MACS the first MAss
Conditioned 3D hand and object motion Synthesis approach. Our approach is based
on cascaded diffusion models and generates interactions that plausibly adjust
based on the object mass and interaction type. MACS also accepts a manually
drawn 3D object trajectory as input and synthesizes the natural 3D hand motions
conditioned by the object mass. This flexibility enables MACS to be used for
various downstream applications, such as generating synthetic training data for
ML tasks, fast animation of hands for graphics workflows, and generating
character interactions for computer games. We show experimentally that a
small-scale dataset is sufficient for MACS to reasonably generalize across
interpolated and extrapolated object masses unseen during the training.
Furthermore, MACS shows moderate generalization to unseen objects, thanks to
the mass-conditioned contact labels generated by our surface contact synthesis
model ConNet. Our comprehensive user study confirms that the synthesized 3D
hand-object interactions are highly plausible and realistic.
658a5e83bb5062b4de192ad5
Soshi Shimada
658a5e83bb5062b4de192ad6
Franziska Mueller
658a5e83bb5062b4de192ad7
Jan Bednarik
658a5e83bb5062b4de192ad8
Bardia Doosti
658a5e83bb5062b4de192ad9
Bernd Bickel
658a5e83bb5062b4de192ada
Danhang Tang
658a5e83bb5062b4de192adb
Vladislav Golyanik
658a5e83bb5062b4de192adc
Jonathan Taylor
658a5e83bb5062b4de192add
Christian Theobalt
658a5e83bb5062b4de192ade
Thabo Beeler
MACS: Mass Conditioned 3D Hand and Object Motion Synthesis
The physical properties of an object, such as mass, significantly affect how
we manipulate it with our hands. Surprisingly, this aspect has so far been
neglected in prior work on 3D motion synthesis. To improve the naturalness of
the synthesized 3D hand object motions, this work proposes MACS the first MAss
Conditioned 3D hand and object motion Synthesis approach. Our approach is based
on cascaded diffusion models and generates interactions that plausibly adjust
based on the object mass and interaction type. MACS also accepts a manually
drawn 3D object trajectory as input and synthesizes the natural 3D hand motions
conditioned by the object mass. This flexibility enables MACS to be used for
various downstream applications, such as generating synthetic training data for
ML tasks, fast animation of hands for graphics workflows, and generating
character interactions for computer games. We show experimentally that a
small-scale dataset is sufficient for MACS to reasonably generalize across
interpolated and extrapolated object masses unseen during the training.
Furthermore, MACS shows moderate generalization to unseen objects, thanks to
the mass-conditioned contact labels generated by our surface contact synthesis
model ConNet. Our comprehensive user study confirms that the synthesized 3D
hand-object interactions are highly plausible and realistic.
MACS, using cascaded diffusion models, synthesizes realistic 3D hand-object interactions conditioned on mass, supporting various applications like training data generation and computer games.
cascaded diffusion models surface contact synthesis model ConNet