title: Target-Aware Video Diffusion Models
summary: We present a target-aware video diffusion model that generates videos from an
  input image in which an actor interacts with a specified target while
  performing a desired action. The target is defined by a segmentation mask and
  the desired action is described via a text prompt. Unlike existing controllable
  image-to-video diffusion models that often rely on dense structural or motion
  cues to guide the actor's movements toward the target, our target-aware model
  requires only a simple mask to indicate the target, leveraging the
  generalization capabilities of pretrained models to produce plausible actions.
  This makes our method particularly effective for human-object interaction (HOI)
  scenarios, where providing precise action guidance is challenging, and further
  enables the use of video diffusion models for high-level action planning in
  applications such as robotics. We build our target-aware model by extending a
  baseline model to incorporate the target mask as an additional input. To
  enforce target awareness, we introduce a special token that encodes the
  target's spatial information within the text prompt. We then fine-tune the
  model with our curated dataset using a novel cross-attention loss that aligns
  the cross-attention maps associated with this token with the input target mask.
  To further improve performance, we selectively apply this loss to the most
  semantically relevant transformer blocks and attention regions. Experimental
  results show that our target-aware model outperforms existing solutions in
  generating videos where actors interact accurately with the specified targets.
  We further demonstrate its efficacy in two downstream applications: video
  content creation and zero-shot 3D HOI motion synthesis.
paper_authors._id: 67ee9fb0ffcd7f6fd1bfffbc
paper_authors.name: Taeksoo Kim
paper_authors._id: 67ee9fb0ffcd7f6fd1bfffbd
paper_authors.name: Hanbyul Joo
paper_title: Target-Aware Video Diffusion Models
paper_summary: We present a target-aware video diffusion model that generates videos from an
  input image in which an actor interacts with a specified target while
  performing a desired action. The target is defined by a segmentation mask and
  the desired action is described via a text prompt. Unlike existing controllable
  image-to-video diffusion models that often rely on dense structural or motion
  cues to guide the actor's movements toward the target, our target-aware model
  requires only a simple mask to indicate the target, leveraging the
  generalization capabilities of pretrained models to produce plausible actions.
  This makes our method particularly effective for human-object interaction (HOI)
  scenarios, where providing precise action guidance is challenging, and further
  enables the use of video diffusion models for high-level action planning in
  applications such as robotics. We build our target-aware model by extending a
  baseline model to incorporate the target mask as an additional input. To
  enforce target awareness, we introduce a special token that encodes the
  target's spatial information within the text prompt. We then fine-tune the
  model with our curated dataset using a novel cross-attention loss that aligns
  the cross-attention maps associated with this token with the input target mask.
  To further improve performance, we selectively apply this loss to the most
  semantically relevant transformer blocks and attention regions. Experimental
  results show that our target-aware model outperforms existing solutions in
  generating videos where actors interact accurately with the specified targets.
  We further demonstrate its efficacy in two downstream applications: video
  content creation and zero-shot 3D HOI motion synthesis.
paper_ai_summary: A target-aware video diffusion model generates accurate actor-target interactions from images using segmentation masks and text prompts, outperforming existing methods in HOI scenarios and video applications.
paper_ai_keywords: video diffusion model segmentation mask text prompt controllable image-to-video diffusion models human-object interaction HOI scenarios robotics cross-attention loss transformer blocks attention regions zero-shot 3D HOI motion synthesis