SViMo: Synchronized Diffusion for Video and Motion Generation in
  Hand-object Interaction Scenarios
Hand-Object Interaction (HOI) generation has significant application
potential. However, current 3D HOI motion generation approaches heavily rely on
predefined 3D object models and lab-captured motion data, limiting
generalization capabilities. Meanwhile, HOI video generation methods prioritize
pixel-level visual fidelity, often sacrificing physical plausibility.
Recognizing that visual appearance and motion patterns share fundamental
physical laws in the real world, we propose a novel framework that combines
visual priors and dynamic constraints within a synchronized diffusion process
to generate the HOI video and motion simultaneously. To integrate the
heterogeneous semantics, appearance, and motion features, our method implements
tri-modal adaptive modulation for feature aligning, coupled with 3D
full-attention for modeling inter- and intra-modal dependencies. Furthermore,
we introduce a vision-aware 3D interaction diffusion model that generates
explicit 3D interaction sequences directly from the synchronized diffusion
outputs, then feeds them back to establish a closed-loop feedback cycle. This
architecture eliminates dependencies on predefined object models or explicit
pose guidance while significantly enhancing video-motion consistency.
Experimental results demonstrate our method's superiority over state-of-the-art
approaches in generating high-fidelity, dynamically plausible HOI sequences,
with notable generalization capabilities in unseen real-world scenarios.
Project page at https://github.com/Droliven/SViMo\_project.
6841bc292852c1d7d4ab7c43
Lingwei Dang
claimed_verified
2025-06-05T17:08:01.386Z
62ebd791fee90fca4742ead8
https://cdn-avatars.huggingface.co/v1/production/uploads/62ebd791fee90fca4742ead8/8-iJYS9Wk41l6JTGtJrNf.jpeg
levon dang
user
levondang
6841bc292852c1d7d4ab7c44
Ruizhi Shao
6841bc292852c1d7d4ab7c45
Hongwen Zhang
6841bc292852c1d7d4ab7c46
Wei Min
6841bc292852c1d7d4ab7c47
Yebin Liu
6841bc292852c1d7d4ab7c48
Qingyao Wu
SViMo: Synchronized Diffusion for Video and Motion Generation in
  Hand-object Interaction Scenarios
Hand-Object Interaction (HOI) generation has significant application
potential. However, current 3D HOI motion generation approaches heavily rely on
predefined 3D object models and lab-captured motion data, limiting
generalization capabilities. Meanwhile, HOI video generation methods prioritize
pixel-level visual fidelity, often sacrificing physical plausibility.
Recognizing that visual appearance and motion patterns share fundamental
physical laws in the real world, we propose a novel framework that combines
visual priors and dynamic constraints within a synchronized diffusion process
to generate the HOI video and motion simultaneously. To integrate the
heterogeneous semantics, appearance, and motion features, our method implements
tri-modal adaptive modulation for feature aligning, coupled with 3D
full-attention for modeling inter- and intra-modal dependencies. Furthermore,
we introduce a vision-aware 3D interaction diffusion model that generates
explicit 3D interaction sequences directly from the synchronized diffusion
outputs, then feeds them back to establish a closed-loop feedback cycle. This
architecture eliminates dependencies on predefined object models or explicit
pose guidance while significantly enhancing video-motion consistency.
Experimental results demonstrate our method's superiority over state-of-the-art
approaches in generating high-fidelity, dynamically plausible HOI sequences,
with notable generalization capabilities in unseen real-world scenarios.
Project page at https://github.com/Droliven/SViMo\_project.
A framework combining visual priors and dynamic constraints within a synchronized diffusion process generates HOI video and motion simultaneously, enhancing video-motion consistency and generalization.
synchronized diffusion process tri-modal adaptive modulation 3D full-attention vision-aware 3D interaction diffusion model HOI video generation HOI motion generation