title: MotiF: Making Text Count in Image Animation with Motion Focal Loss
summary: Text-Image-to-Video (TI2V) generation aims to generate a video from an image
  following a text description, which is also referred to as text-guided image
  animation. Most existing methods struggle to generate videos that align well
  with the text prompts, particularly when motion is specified. To overcome this
  limitation, we introduce MotiF, a simple yet effective approach that directs
  the model's learning to the regions with more motion, thereby improving the
  text alignment and motion generation. We use optical flow to generate a motion
  heatmap and weight the loss according to the intensity of the motion. This
  modified objective leads to noticeable improvements and complements existing
  methods that utilize motion priors as model inputs. Additionally, due to the
  lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V
  Bench, a dataset consists of 320 image-text pairs for robust evaluation. We
  present a human evaluation protocol that asks the annotators to select an
  overall preference between two videos followed by their justifications. Through
  a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced
  models, achieving an average preference of 72%. The TI2V Bench is released in
  https://wang-sj16.github.io/motif/.
paper_authors._id: 676c4a04295f85d93ef6d6b6
paper_authors.name: Shijie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:32:14.529Z
paper_authors._id: 6438c702c04b3b996ea702fb
paper_authors.avatarUrl: /avatars/09a386c4e810193d2b1f0b7799bc8270.svg
paper_authors.fullname: Shijie Wang
paper_authors.type: user
paper_authors.user: wang-sj16
paper_authors._id: 676c4a04295f85d93ef6d6b7
paper_authors.name: Samaneh Azadi
paper_authors._id: 676c4a04295f85d93ef6d6b8
paper_authors.name: Rohit Girdhar
paper_authors._id: 676c4a04295f85d93ef6d6b9
paper_authors.name: Saketh Rambhatla
paper_authors._id: 676c4a04295f85d93ef6d6ba
paper_authors.name: Chen Sun
paper_authors._id: 676c4a04295f85d93ef6d6bb
paper_authors.name: Xi Yin
paper_title: MotiF: Making Text Count in Image Animation with Motion Focal Loss
paper_summary: Text-Image-to-Video (TI2V) generation aims to generate a video from an image
  following a text description, which is also referred to as text-guided image
  animation. Most existing methods struggle to generate videos that align well
  with the text prompts, particularly when motion is specified. To overcome this
  limitation, we introduce MotiF, a simple yet effective approach that directs
  the model's learning to the regions with more motion, thereby improving the
  text alignment and motion generation. We use optical flow to generate a motion
  heatmap and weight the loss according to the intensity of the motion. This
  modified objective leads to noticeable improvements and complements existing
  methods that utilize motion priors as model inputs. Additionally, due to the
  lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V
  Bench, a dataset consists of 320 image-text pairs for robust evaluation. We
  present a human evaluation protocol that asks the annotators to select an
  overall preference between two videos followed by their justifications. Through
  a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced
  models, achieving an average preference of 72%. The TI2V Bench is released in
  https://wang-sj16.github.io/motif/.
paper_ai_summary: MotiF improves text-guided image-to-video generation by focusing on motion regions, enhancing text alignment and motion quality using a motion heatmap and a newly proposed benchmark dataset.
paper_ai_keywords: text-image-to-video generation text-guided image animation motion heatmap optical flow MotiF TI2V Bench human evaluation protocol