title: MACS: Mass Conditioned 3D Hand and Object Motion Synthesis
summary: The physical properties of an object, such as mass, significantly affect how
  we manipulate it with our hands. Surprisingly, this aspect has so far been
  neglected in prior work on 3D motion synthesis. To improve the naturalness of
  the synthesized 3D hand object motions, this work proposes MACS the first MAss
  Conditioned 3D hand and object motion Synthesis approach. Our approach is based
  on cascaded diffusion models and generates interactions that plausibly adjust
  based on the object mass and interaction type. MACS also accepts a manually
  drawn 3D object trajectory as input and synthesizes the natural 3D hand motions
  conditioned by the object mass. This flexibility enables MACS to be used for
  various downstream applications, such as generating synthetic training data for
  ML tasks, fast animation of hands for graphics workflows, and generating
  character interactions for computer games. We show experimentally that a
  small-scale dataset is sufficient for MACS to reasonably generalize across
  interpolated and extrapolated object masses unseen during the training.
  Furthermore, MACS shows moderate generalization to unseen objects, thanks to
  the mass-conditioned contact labels generated by our surface contact synthesis
  model ConNet. Our comprehensive user study confirms that the synthesized 3D
  hand-object interactions are highly plausible and realistic.
paper_authors._id: 658a5e83bb5062b4de192ad5
paper_authors.name: Soshi Shimada
paper_authors._id: 658a5e83bb5062b4de192ad6
paper_authors.name: Franziska Mueller
paper_authors._id: 658a5e83bb5062b4de192ad7
paper_authors.name: Jan Bednarik
paper_authors._id: 658a5e83bb5062b4de192ad8
paper_authors.name: Bardia Doosti
paper_authors._id: 658a5e83bb5062b4de192ad9
paper_authors.name: Bernd Bickel
paper_authors._id: 658a5e83bb5062b4de192ada
paper_authors.name: Danhang Tang
paper_authors._id: 658a5e83bb5062b4de192adb
paper_authors.name: Vladislav Golyanik
paper_authors._id: 658a5e83bb5062b4de192adc
paper_authors.name: Jonathan Taylor
paper_authors._id: 658a5e83bb5062b4de192add
paper_authors.name: Christian Theobalt
paper_authors._id: 658a5e83bb5062b4de192ade
paper_authors.name: Thabo Beeler
paper_title: MACS: Mass Conditioned 3D Hand and Object Motion Synthesis
paper_summary: The physical properties of an object, such as mass, significantly affect how
  we manipulate it with our hands. Surprisingly, this aspect has so far been
  neglected in prior work on 3D motion synthesis. To improve the naturalness of
  the synthesized 3D hand object motions, this work proposes MACS the first MAss
  Conditioned 3D hand and object motion Synthesis approach. Our approach is based
  on cascaded diffusion models and generates interactions that plausibly adjust
  based on the object mass and interaction type. MACS also accepts a manually
  drawn 3D object trajectory as input and synthesizes the natural 3D hand motions
  conditioned by the object mass. This flexibility enables MACS to be used for
  various downstream applications, such as generating synthetic training data for
  ML tasks, fast animation of hands for graphics workflows, and generating
  character interactions for computer games. We show experimentally that a
  small-scale dataset is sufficient for MACS to reasonably generalize across
  interpolated and extrapolated object masses unseen during the training.
  Furthermore, MACS shows moderate generalization to unseen objects, thanks to
  the mass-conditioned contact labels generated by our surface contact synthesis
  model ConNet. Our comprehensive user study confirms that the synthesized 3D
  hand-object interactions are highly plausible and realistic.
paper_ai_keywords: cascaded diffusion models multimodal MAss Conditioned 3D hand and object motion Synthesis (MACS) 3D object trajectory 3D hand motions generalization surface contact synthesis model (ConNet)