title: Controllable Human-Object Interaction Synthesis
summary: Synthesizing semantic-aware, long-horizon, human-object interaction is
  critical to simulate realistic human behaviors. In this work, we address the
  challenging problem of generating synchronized object motion and human motion
  guided by language descriptions in 3D scenes. We propose Controllable
  Human-Object Interaction Synthesis (CHOIS), an approach that generates object
  motion and human motion simultaneously using a conditional diffusion model
  given a language description, initial object and human states, and sparse
  object waypoints. While language descriptions inform style and intent,
  waypoints ground the motion in the scene and can be effectively extracted using
  high-level planning methods. Naively applying a diffusion model fails to
  predict object motion aligned with the input waypoints and cannot ensure the
  realism of interactions that require precise hand-object contact and
  appropriate contact grounded by the floor. To overcome these problems, we
  introduce an object geometry loss as additional supervision to improve the
  matching between generated object motion and input object waypoints. In
  addition, we design guidance terms to enforce contact constraints during the
  sampling process of the trained diffusion model.
paper_authors._id: 65729540cca2247f3f2c59f4
paper_authors.name: Jiaman Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:27:54.335Z
paper_authors._id: 653b7d1732bd4db35d9ada4c
paper_authors.avatarUrl: /avatars/b829135b7b36c07e5f47c343f8b82594.svg
paper_authors.fullname: Jiaman Li
paper_authors.type: user
paper_authors.user: jiaman
paper_authors._id: 65729540cca2247f3f2c59f5
paper_authors.name: Alexander Clegg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:28:02.186Z
paper_authors._id: 63eabb40a332618465d4f698
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676327808512-63eabb40a332618465d4f698.jpeg
paper_authors.fullname: Alexander Clegg
paper_authors.type: user
paper_authors.user: aclegg3
paper_authors._id: 65729540cca2247f3f2c59f6
paper_authors.name: Roozbeh Mottaghi
paper_authors._id: 65729540cca2247f3f2c59f7
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:28:27.192Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 65729540cca2247f3f2c59f8
paper_authors.name: Xavier Puig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:28:33.829Z
paper_authors._id: 64669f6afc452cf123c0a0b6
paper_authors.avatarUrl: /avatars/9dec7a63d38bfd8606a468b0eb77c6e5.svg
paper_authors.fullname: Xavier Puig
paper_authors.type: user
paper_authors.user: xavierpuigf
paper_authors._id: 65729540cca2247f3f2c59f9
paper_authors.name: C. Karen Liu
paper_title: Controllable Human-Object Interaction Synthesis
paper_summary: Synthesizing semantic-aware, long-horizon, human-object interaction is
  critical to simulate realistic human behaviors. In this work, we address the
  challenging problem of generating synchronized object motion and human motion
  guided by language descriptions in 3D scenes. We propose Controllable
  Human-Object Interaction Synthesis (CHOIS), an approach that generates object
  motion and human motion simultaneously using a conditional diffusion model
  given a language description, initial object and human states, and sparse
  object waypoints. While language descriptions inform style and intent,
  waypoints ground the motion in the scene and can be effectively extracted using
  high-level planning methods. Naively applying a diffusion model fails to
  predict object motion aligned with the input waypoints and cannot ensure the
  realism of interactions that require precise hand-object contact and
  appropriate contact grounded by the floor. To overcome these problems, we
  introduce an object geometry loss as additional supervision to improve the
  matching between generated object motion and input object waypoints. In
  addition, we design guidance terms to enforce contact constraints during the
  sampling process of the trained diffusion model.
paper_ai_keywords: conditional diffusion model language descriptions object waypoints object geometry loss guidance terms contact constraints