title: MotionBooth: Motion-Aware Customized Text-to-Video Generation
summary: In this work, we present MotionBooth, an innovative framework designed for
  animating customized subjects with precise control over both object and camera
  movements. By leveraging a few images of a specific object, we efficiently
  fine-tune a text-to-video model to capture the object's shape and attributes
  accurately. Our approach presents subject region loss and video preservation
  loss to enhance the subject's learning performance, along with a subject token
  cross-attention loss to integrate the customized subject with motion control
  signals. Additionally, we propose training-free techniques for managing subject
  and camera motions during inference. In particular, we utilize cross-attention
  map manipulation to govern subject motion and introduce a novel latent shift
  module for camera movement control as well. MotionBooth excels in preserving
  the appearance of subjects while simultaneously controlling the motions in
  generated videos. Extensive quantitative and qualitative evaluations
  demonstrate the superiority and effectiveness of our method. Our project page
  is at https://jianzongwu.github.io/projects/motionbooth
paper_authors._id: 667bacafeeef441d207e8007
paper_authors.name: Jianzong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:12:19.381Z
paper_authors._id: 657a6eed1ccc3c2a5ea7b585
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RIQIF-JJdNI0SwJEq_9z7.jpeg
paper_authors.fullname: Jianzong Wu
paper_authors.type: user
paper_authors.user: jianzongwu
paper_authors._id: 667bacafeeef441d207e8008
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:12:37.354Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 667bacafeeef441d207e8009
paper_authors.name: Yanhong Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:03:16.354Z
paper_authors._id: 63d4b843df01ef426a0f79fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
paper_authors.fullname: Yanhong Zeng
paper_authors.type: user
paper_authors.user: zengyh1900
paper_authors._id: 667bacafeeef441d207e800a
paper_authors.name: Jiangning Zhang
paper_authors._id: 667bacafeeef441d207e800b
paper_authors.name: Qianyu Zhou
paper_authors._id: 667bacafeeef441d207e800c
paper_authors.name: Yining Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T13:13:50.314Z
paper_authors._id: 6570542c91da380dfbdd2d36
paper_authors.avatarUrl: /avatars/cf3e7f7c5bf790f93fc6b4cb5d72c915.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ANTiSOC1AL
paper_authors._id: 667bacafeeef441d207e800d
paper_authors.name: Yunhai Tong
paper_authors._id: 667bacafeeef441d207e800e
paper_authors.name: Kai Chen
paper_title: MotionBooth: Motion-Aware Customized Text-to-Video Generation
paper_summary: In this work, we present MotionBooth, an innovative framework designed for
  animating customized subjects with precise control over both object and camera
  movements. By leveraging a few images of a specific object, we efficiently
  fine-tune a text-to-video model to capture the object's shape and attributes
  accurately. Our approach presents subject region loss and video preservation
  loss to enhance the subject's learning performance, along with a subject token
  cross-attention loss to integrate the customized subject with motion control
  signals. Additionally, we propose training-free techniques for managing subject
  and camera motions during inference. In particular, we utilize cross-attention
  map manipulation to govern subject motion and introduce a novel latent shift
  module for camera movement control as well. MotionBooth excels in preserving
  the appearance of subjects while simultaneously controlling the motions in
  generated videos. Extensive quantitative and qualitative evaluations
  demonstrate the superiority and effectiveness of our method. Our project page
  is at https://jianzongwu.github.io/projects/motionbooth
paper_ai_keywords: text-to-video model subject region loss video preservation loss subject token cross-attention loss cross-attention map manipulation latent shift module