title: MotionClone: Training-Free Motion Cloning for Controllable Video
    Generation
summary: Motion-based controllable text-to-video generation involves motions to
  control the video generation. Previous methods typically require the training
  of models to encode motion cues or the fine-tuning of video diffusion models.
  However, these approaches often result in suboptimal motion generation when
  applied outside the trained domain. In this work, we propose MotionClone, a
  training-free framework that enables motion cloning from a reference video to
  control text-to-video generation. We employ temporal attention in video
  inversion to represent the motions in the reference video and introduce primary
  temporal-attention guidance to mitigate the influence of noisy or very subtle
  motions within the attention weights. Furthermore, to assist the generation
  model in synthesizing reasonable spatial relationships and enhance its
  prompt-following capability, we propose a location-aware semantic guidance
  mechanism that leverages the coarse location of the foreground from the
  reference video and original classifier-free guidance features to guide the
  video generation. Extensive experiments demonstrate that MotionClone exhibits
  proficiency in both global camera motion and local object motion, with notable
  superiority in terms of motion fidelity, textual alignment, and temporal
  consistency.
paper_authors._id: 666a7153be667d2b1db1d12a
paper_authors.name: Pengyang Ling
paper_authors._id: 666a7153be667d2b1db1d12b
paper_authors.name: Jiazi Bu
paper_authors._id: 666a7153be667d2b1db1d12c
paper_authors.name: Pan Zhang
paper_authors._id: 666a7153be667d2b1db1d12d
paper_authors.name: Xiaoyi Dong
paper_authors._id: 666a7153be667d2b1db1d12e
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:30:50.778Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 666a7153be667d2b1db1d12f
paper_authors.name: Tong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:24:20.409Z
paper_authors._id: 6310929fda0aaad2f70397a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310929fda0aaad2f70397a1/YoHeYZezmCOXwL-qsiZOc.png
paper_authors.fullname: Tong Wu
paper_authors.type: user
paper_authors.user: wutong16
paper_authors._id: 666a7153be667d2b1db1d130
paper_authors.name: Huaian Chen
paper_authors._id: 666a7153be667d2b1db1d131
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T19:29:31.823Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_authors._id: 666a7153be667d2b1db1d132
paper_authors.name: Yi Jin
paper_title: MotionClone: Training-Free Motion Cloning for Controllable Video
    Generation
paper_summary: Motion-based controllable text-to-video generation involves motions to
  control the video generation. Previous methods typically require the training
  of models to encode motion cues or the fine-tuning of video diffusion models.
  However, these approaches often result in suboptimal motion generation when
  applied outside the trained domain. In this work, we propose MotionClone, a
  training-free framework that enables motion cloning from a reference video to
  control text-to-video generation. We employ temporal attention in video
  inversion to represent the motions in the reference video and introduce primary
  temporal-attention guidance to mitigate the influence of noisy or very subtle
  motions within the attention weights. Furthermore, to assist the generation
  model in synthesizing reasonable spatial relationships and enhance its
  prompt-following capability, we propose a location-aware semantic guidance
  mechanism that leverages the coarse location of the foreground from the
  reference video and original classifier-free guidance features to guide the
  video generation. Extensive experiments demonstrate that MotionClone exhibits
  proficiency in both global camera motion and local object motion, with notable
  superiority in terms of motion fidelity, textual alignment, and temporal
  consistency.
paper_ai_keywords: MotionClone motion cloning video inversion temporal attention primary temporal-attention guidance location-aware semantic guidance coarse location foreground original classifier-free guidance features video generation motion fidelity textual alignment temporal consistency