MotionBooth: Motion-Aware Customized Text-to-Video Generation
In this work, we present MotionBooth, an innovative framework designed for
animating customized subjects with precise control over both object and camera
movements. By leveraging a few images of a specific object, we efficiently
fine-tune a text-to-video model to capture the object's shape and attributes
accurately. Our approach presents subject region loss and video preservation
loss to enhance the subject's learning performance, along with a subject token
cross-attention loss to integrate the customized subject with motion control
signals. Additionally, we propose training-free techniques for managing subject
and camera motions during inference. In particular, we utilize cross-attention
map manipulation to govern subject motion and introduce a novel latent shift
module for camera movement control as well. MotionBooth excels in preserving
the appearance of subjects while simultaneously controlling the motions in
generated videos. Extensive quantitative and qualitative evaluations
demonstrate the superiority and effectiveness of our method. Our project page
is at https://jianzongwu.github.io/projects/motionbooth
667bacafeeef441d207e8007
Jianzong Wu
admin_assigned
2024-06-26T13:12:19.381Z
657a6eed1ccc3c2a5ea7b585
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RIQIF-JJdNI0SwJEq_9z7.jpeg
Jianzong Wu
user
jianzongwu
667bacafeeef441d207e8008
Xiangtai Li
admin_assigned
2024-06-26T13:12:37.354Z
63958b4414513eaf9029ebf1
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
Xiangtai Li
user
LXT
667bacafeeef441d207e8009
Yanhong Zeng
claimed_verified
2024-06-27T09:03:16.354Z
63d4b843df01ef426a0f79fb
https://cdn-avatars.huggingface.co/v1/production/uploads/1676365795587-63d4b843df01ef426a0f79fb.jpeg
Yanhong Zeng
user
zengyh1900
667bacafeeef441d207e800a
Jiangning Zhang
667bacafeeef441d207e800b
Qianyu Zhou
667bacafeeef441d207e800c
Yining Li
admin_assigned
2024-06-26T13:13:50.314Z
6570542c91da380dfbdd2d36
/avatars/cf3e7f7c5bf790f93fc6b4cb5d72c915.svg
Yining Li
user
ANTiSOC1AL
667bacafeeef441d207e800d
Yunhai Tong
667bacafeeef441d207e800e
Kai Chen
MotionBooth: Motion-Aware Customized Text-to-Video Generation
In this work, we present MotionBooth, an innovative framework designed for
animating customized subjects with precise control over both object and camera
movements. By leveraging a few images of a specific object, we efficiently
fine-tune a text-to-video model to capture the object's shape and attributes
accurately. Our approach presents subject region loss and video preservation
loss to enhance the subject's learning performance, along with a subject token
cross-attention loss to integrate the customized subject with motion control
signals. Additionally, we propose training-free techniques for managing subject
and camera motions during inference. In particular, we utilize cross-attention
map manipulation to govern subject motion and introduce a novel latent shift
module for camera movement control as well. MotionBooth excels in preserving
the appearance of subjects while simultaneously controlling the motions in
generated videos. Extensive quantitative and qualitative evaluations
demonstrate the superiority and effectiveness of our method. Our project page
is at https://jianzongwu.github.io/projects/motionbooth
MotionBooth fine-tunes a text-to-video model using a few images to animate customized subjects with precise control over object and camera movements, enhancing performance with subject region loss, video preservation loss, and subject token cross-attention loss.
text-to-video model subject region loss video preservation loss subject token cross-attention loss cross-attention map manipulation latent shift module