title: SViMo: Synchronized Diffusion for Video and Motion Generation in
    Hand-object Interaction Scenarios
summary: Hand-Object Interaction (HOI) generation has significant application
  potential. However, current 3D HOI motion generation approaches heavily rely on
  predefined 3D object models and lab-captured motion data, limiting
  generalization capabilities. Meanwhile, HOI video generation methods prioritize
  pixel-level visual fidelity, often sacrificing physical plausibility.
  Recognizing that visual appearance and motion patterns share fundamental
  physical laws in the real world, we propose a novel framework that combines
  visual priors and dynamic constraints within a synchronized diffusion process
  to generate the HOI video and motion simultaneously. To integrate the
  heterogeneous semantics, appearance, and motion features, our method implements
  tri-modal adaptive modulation for feature aligning, coupled with 3D
  full-attention for modeling inter- and intra-modal dependencies. Furthermore,
  we introduce a vision-aware 3D interaction diffusion model that generates
  explicit 3D interaction sequences directly from the synchronized diffusion
  outputs, then feeds them back to establish a closed-loop feedback cycle. This
  architecture eliminates dependencies on predefined object models or explicit
  pose guidance while significantly enhancing video-motion consistency.
  Experimental results demonstrate our method's superiority over state-of-the-art
  approaches in generating high-fidelity, dynamically plausible HOI sequences,
  with notable generalization capabilities in unseen real-world scenarios.
  Project page at https://github.com/Droliven/SViMo\_project.
paper_authors._id: 6841bc292852c1d7d4ab7c43
paper_authors.name: Lingwei Dang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T17:08:01.386Z
paper_authors._id: 62ebd791fee90fca4742ead8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62ebd791fee90fca4742ead8/8-iJYS9Wk41l6JTGtJrNf.jpeg
paper_authors.fullname: levon dang
paper_authors.type: user
paper_authors.user: levondang
paper_authors._id: 6841bc292852c1d7d4ab7c44
paper_authors.name: Ruizhi Shao
paper_authors._id: 6841bc292852c1d7d4ab7c45
paper_authors.name: Hongwen Zhang
paper_authors._id: 6841bc292852c1d7d4ab7c46
paper_authors.name: Wei Min
paper_authors._id: 6841bc292852c1d7d4ab7c47
paper_authors.name: Yebin Liu
paper_authors._id: 6841bc292852c1d7d4ab7c48
paper_authors.name: Qingyao Wu
paper_title: SViMo: Synchronized Diffusion for Video and Motion Generation in
    Hand-object Interaction Scenarios
paper_summary: Hand-Object Interaction (HOI) generation has significant application
  potential. However, current 3D HOI motion generation approaches heavily rely on
  predefined 3D object models and lab-captured motion data, limiting
  generalization capabilities. Meanwhile, HOI video generation methods prioritize
  pixel-level visual fidelity, often sacrificing physical plausibility.
  Recognizing that visual appearance and motion patterns share fundamental
  physical laws in the real world, we propose a novel framework that combines
  visual priors and dynamic constraints within a synchronized diffusion process
  to generate the HOI video and motion simultaneously. To integrate the
  heterogeneous semantics, appearance, and motion features, our method implements
  tri-modal adaptive modulation for feature aligning, coupled with 3D
  full-attention for modeling inter- and intra-modal dependencies. Furthermore,
  we introduce a vision-aware 3D interaction diffusion model that generates
  explicit 3D interaction sequences directly from the synchronized diffusion
  outputs, then feeds them back to establish a closed-loop feedback cycle. This
  architecture eliminates dependencies on predefined object models or explicit
  pose guidance while significantly enhancing video-motion consistency.
  Experimental results demonstrate our method's superiority over state-of-the-art
  approaches in generating high-fidelity, dynamically plausible HOI sequences,
  with notable generalization capabilities in unseen real-world scenarios.
  Project page at https://github.com/Droliven/SViMo\_project.
paper_ai_summary: A framework combining visual priors and dynamic constraints within a synchronized diffusion process generates HOI video and motion simultaneously, enhancing video-motion consistency and generalization.
paper_ai_keywords: synchronized diffusion process tri-modal adaptive modulation 3D full-attention vision-aware 3D interaction diffusion model HOI video generation HOI motion generation