title: LivePhoto: Real Image Animation with Text-guided Motion Control
summary: Despite the recent progress in text-to-video generation, existing studies
  usually overlook the issue that only spatial contents but not temporal motions
  in synthesized videos are under the control of text. Towards such a challenge,
  this work presents a practical system, named LivePhoto, which allows users to
  animate an image of their interest with text descriptions. We first establish a
  strong baseline that helps a well-learned text-to-image generator (i.e., Stable
  Diffusion) take an image as a further input. We then equip the improved
  generator with a motion module for temporal modeling and propose a carefully
  designed training pipeline to better link texts and motions. In particular,
  considering the facts that (1) text can only describe motions roughly (e.g.,
  regardless of the moving speed) and (2) text may include both content and
  motion descriptions, we introduce a motion intensity estimation module as well
  as a text re-weighting module to reduce the ambiguity of text-to-motion
  mapping. Empirical evidence suggests that our approach is capable of well
  decoding motion-related textual instructions into videos, such as actions,
  camera movements, or even conjuring new contents from thin air (e.g., pouring
  water into an empty glass). Interestingly, thanks to the proposed intensity
  learning mechanism, our system offers users an additional control signal (i.e.,
  the motion intensity) besides text for video customization.
paper_authors._id: 656fdb0f2cf29b89e91583df
paper_authors.name: Xi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T08:30:00.996Z
paper_authors._id: 644a1b6401e18bf93a6f45c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
paper_authors.fullname: xichen
paper_authors.type: user
paper_authors.user: xichenhku
paper_authors._id: 656fdb0f2cf29b89e91583e0
paper_authors.name: Zhiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T12:00:10.882Z
paper_authors._id: 6479925ab77e18dbf640bd67
paper_authors.avatarUrl: /avatars/bb52ecd22ca4b49157f8668be35409e7.svg
paper_authors.fullname: Zhiheng Liu
paper_authors.type: user
paper_authors.user: Johanan0528
paper_authors._id: 656fdb0f2cf29b89e91583e1
paper_authors.name: Mengting Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T12:00:16.991Z
paper_authors._id: 641073eb928400b416444af5
paper_authors.avatarUrl: /avatars/0eed14a8929682f182e474f3693fc442.svg
paper_authors.fullname: Chenmengting
paper_authors.type: user
paper_authors.user: MengTingChen
paper_authors._id: 656fdb0f2cf29b89e91583e2
paper_authors.name: Yutong Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-19T16:01:27.787Z
paper_authors._id: 64a54e468cfaa458bd6844bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
paper_authors.fullname: Yutong Feng
paper_authors.type: user
paper_authors.user: yutongfeng
paper_authors._id: 656fdb0f2cf29b89e91583e3
paper_authors.name: Yu Liu
paper_authors._id: 656fdb0f2cf29b89e91583e4
paper_authors.name: Yujun Shen
paper_authors._id: 656fdb0f2cf29b89e91583e5
paper_authors.name: Hengshuang Zhao
paper_title: LivePhoto: Real Image Animation with Text-guided Motion Control
paper_summary: Despite the recent progress in text-to-video generation, existing studies
  usually overlook the issue that only spatial contents but not temporal motions
  in synthesized videos are under the control of text. Towards such a challenge,
  this work presents a practical system, named LivePhoto, which allows users to
  animate an image of their interest with text descriptions. We first establish a
  strong baseline that helps a well-learned text-to-image generator (i.e., Stable
  Diffusion) take an image as a further input. We then equip the improved
  generator with a motion module for temporal modeling and propose a carefully
  designed training pipeline to better link texts and motions. In particular,
  considering the facts that (1) text can only describe motions roughly (e.g.,
  regardless of the moving speed) and (2) text may include both content and
  motion descriptions, we introduce a motion intensity estimation module as well
  as a text re-weighting module to reduce the ambiguity of text-to-motion
  mapping. Empirical evidence suggests that our approach is capable of well
  decoding motion-related textual instructions into videos, such as actions,
  camera movements, or even conjuring new contents from thin air (e.g., pouring
  water into an empty glass). Interestingly, thanks to the proposed intensity
  learning mechanism, our system offers users an additional control signal (i.e.,
  the motion intensity) besides text for video customization.
paper_ai_keywords: text-to-video generation spatial contents temporal motions text-to-image generator Stable Diffusion motion module temporal modeling training pipeline motion intensity estimation module text re-weighting module motion intensity video customization