LivePhoto: Real Image Animation with Text-guided Motion Control
Despite the recent progress in text-to-video generation, existing studies
usually overlook the issue that only spatial contents but not temporal motions
in synthesized videos are under the control of text. Towards such a challenge,
this work presents a practical system, named LivePhoto, which allows users to
animate an image of their interest with text descriptions. We first establish a
strong baseline that helps a well-learned text-to-image generator (i.e., Stable
Diffusion) take an image as a further input. We then equip the improved
generator with a motion module for temporal modeling and propose a carefully
designed training pipeline to better link texts and motions. In particular,
considering the facts that (1) text can only describe motions roughly (e.g.,
regardless of the moving speed) and (2) text may include both content and
motion descriptions, we introduce a motion intensity estimation module as well
as a text re-weighting module to reduce the ambiguity of text-to-motion
mapping. Empirical evidence suggests that our approach is capable of well
decoding motion-related textual instructions into videos, such as actions,
camera movements, or even conjuring new contents from thin air (e.g., pouring
water into an empty glass). Interestingly, thanks to the proposed intensity
learning mechanism, our system offers users an additional control signal (i.e.,
the motion intensity) besides text for video customization.
656fdb0f2cf29b89e91583df
Xi Chen
claimed_verified
2023-12-07T08:30:00.996Z
644a1b6401e18bf93a6f45c1
https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
xichen
user
xichenhku
656fdb0f2cf29b89e91583e0
Zhiheng Liu
admin_assigned
2023-12-06T12:00:10.882Z
6479925ab77e18dbf640bd67
/avatars/bb52ecd22ca4b49157f8668be35409e7.svg
Zhiheng Liu
user
Johanan0528
656fdb0f2cf29b89e91583e1
Mengting Chen
admin_assigned
2023-12-06T12:00:16.991Z
641073eb928400b416444af5
/avatars/0eed14a8929682f182e474f3693fc442.svg
Chenmengting
user
MengTingChen
656fdb0f2cf29b89e91583e2
Yutong Feng
claimed_verified
2023-12-19T16:01:27.787Z
64a54e468cfaa458bd6844bf
https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
Yutong Feng
user
fengyutong
656fdb0f2cf29b89e91583e3
Yu Liu
656fdb0f2cf29b89e91583e4
Yujun Shen
656fdb0f2cf29b89e91583e5
Hengshuang Zhao
LivePhoto: Real Image Animation with Text-guided Motion Control
Despite the recent progress in text-to-video generation, existing studies
usually overlook the issue that only spatial contents but not temporal motions
in synthesized videos are under the control of text. Towards such a challenge,
this work presents a practical system, named LivePhoto, which allows users to
animate an image of their interest with text descriptions. We first establish a
strong baseline that helps a well-learned text-to-image generator (i.e., Stable
Diffusion) take an image as a further input. We then equip the improved
generator with a motion module for temporal modeling and propose a carefully
designed training pipeline to better link texts and motions. In particular,
considering the facts that (1) text can only describe motions roughly (e.g.,
regardless of the moving speed) and (2) text may include both content and
motion descriptions, we introduce a motion intensity estimation module as well
as a text re-weighting module to reduce the ambiguity of text-to-motion
mapping. Empirical evidence suggests that our approach is capable of well
decoding motion-related textual instructions into videos, such as actions,
camera movements, or even conjuring new contents from thin air (e.g., pouring
water into an empty glass). Interestingly, thanks to the proposed intensity
learning mechanism, our system offers users an additional control signal (i.e.,
the motion intensity) besides text for video customization.
A text-to-video system, LivePhoto, uses a text-to-image generator enhanced with a motion module to accurately translate textual motion descriptions into videos.
text-to-video generation spatial contents temporal motions text-to-image generator Stable Diffusion motion module temporal modeling training pipeline motion intensity estimation module text re-weighting module motion-related textual instructions video customization