MotionClone: Training-Free Motion Cloning for Controllable Video
  Generation
Motion-based controllable text-to-video generation involves motions to
control the video generation. Previous methods typically require the training
of models to encode motion cues or the fine-tuning of video diffusion models.
However, these approaches often result in suboptimal motion generation when
applied outside the trained domain. In this work, we propose MotionClone, a
training-free framework that enables motion cloning from a reference video to
control text-to-video generation. We employ temporal attention in video
inversion to represent the motions in the reference video and introduce primary
temporal-attention guidance to mitigate the influence of noisy or very subtle
motions within the attention weights. Furthermore, to assist the generation
model in synthesizing reasonable spatial relationships and enhance its
prompt-following capability, we propose a location-aware semantic guidance
mechanism that leverages the coarse location of the foreground from the
reference video and original classifier-free guidance features to guide the
video generation. Extensive experiments demonstrate that MotionClone exhibits
proficiency in both global camera motion and local object motion, with notable
superiority in terms of motion fidelity, textual alignment, and temporal
consistency.
666a7153be667d2b1db1d12a
Pengyang Ling
666a7153be667d2b1db1d12b
Jiazi Bu
666a7153be667d2b1db1d12c
Pan Zhang
666a7153be667d2b1db1d12d
Xiaoyi Dong
666a7153be667d2b1db1d12e
Yuhang Zang
claimed_verified
2024-06-13T09:30:50.778Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
666a7153be667d2b1db1d12f
Tong Wu
claimed_verified
2024-12-11T09:24:20.409Z
6310929fda0aaad2f70397a1
https://cdn-avatars.huggingface.co/v1/production/uploads/6310929fda0aaad2f70397a1/YoHeYZezmCOXwL-qsiZOc.png
Tong Wu
user
wutong16
666a7153be667d2b1db1d130
Huaian Chen
666a7153be667d2b1db1d131
Jiaqi Wang
admin_assigned
2024-06-13T19:29:31.823Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
666a7153be667d2b1db1d132
Yi Jin
MotionClone: Training-Free Motion Cloning for Controllable Video
  Generation
Motion-based controllable text-to-video generation involves motions to
control the video generation. Previous methods typically require the training
of models to encode motion cues or the fine-tuning of video diffusion models.
However, these approaches often result in suboptimal motion generation when
applied outside the trained domain. In this work, we propose MotionClone, a
training-free framework that enables motion cloning from a reference video to
control text-to-video generation. We employ temporal attention in video
inversion to represent the motions in the reference video and introduce primary
temporal-attention guidance to mitigate the influence of noisy or very subtle
motions within the attention weights. Furthermore, to assist the generation
model in synthesizing reasonable spatial relationships and enhance its
prompt-following capability, we propose a location-aware semantic guidance
mechanism that leverages the coarse location of the foreground from the
reference video and original classifier-free guidance features to guide the
video generation. Extensive experiments demonstrate that MotionClone exhibits
proficiency in both global camera motion and local object motion, with notable
superiority in terms of motion fidelity, textual alignment, and temporal
consistency.
MotionClone, a training-free framework, uses motion cloning to control text-to-video generation, yielding superior motion fidelity, textual alignment, and temporal consistency.
motion-based controllable text-to-video generation motion cloning temporal attention video inversion noise mitigation location-aware semantic guidance motion fidelity textual alignment temporal consistency