title: Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video
    Generation
summary: We consider the task of Image-to-Video (I2V) generation, which involves
  transforming static images into realistic video sequences based on a textual
  description. While recent advancements produce photorealistic outputs, they
  frequently struggle to create videos with accurate and consistent object
  motion, especially in multi-object scenarios. To address these limitations, we
  propose a two-stage compositional framework that decomposes I2V generation
  into: (i) An explicit intermediate representation generation stage, followed by
  (ii) A video generation stage that is conditioned on this representation. Our
  key innovation is the introduction of a mask-based motion trajectory as an
  intermediate representation, that captures both semantic object information and
  motion, enabling an expressive but compact representation of motion and
  semantics. To incorporate the learned representation in the second stage, we
  utilize object-level attention objectives. Specifically, we consider a spatial,
  per-object, masked-cross attention objective, integrating object-specific
  prompts into corresponding latent space regions and a masked spatio-temporal
  self-attention objective, ensuring frame-to-frame consistency for each object.
  We evaluate our method on challenging benchmarks with multi-object and
  high-motion scenarios and empirically demonstrate that the proposed method
  achieves state-of-the-art results in temporal coherence, motion realism, and
  text-prompt faithfulness. Additionally, we introduce \benchmark, a new
  challenging benchmark for single-object and multi-object I2V generation, and
  demonstrate our method's superiority on this benchmark. Project page is
  available at https://guyyariv.github.io/TTM/.
paper_authors._id: 677cd67175191fdc79cbcc04
paper_authors.name: Guy Yariv
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T09:37:35.128Z
paper_authors._id: 646d239f4220471ca0c6471c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d239f4220471ca0c6471c/sRwzko8XEUVCkeD7jXceH.jpeg
paper_authors.fullname: Guy Yariv
paper_authors.type: user
paper_authors.user: GuyYariv
paper_authors._id: 677cd67175191fdc79cbcc05
paper_authors.name: Yuval Kirstain
paper_authors._id: 677cd67175191fdc79cbcc06
paper_authors.name: Amit Zohar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:17:40.162Z
paper_authors._id: 64d8cb34505306fcd2fb89c3
paper_authors.avatarUrl: /avatars/4ae9ffe5d494a7d53bc519ff8e402b3e.svg
paper_authors.fullname: Amit Zohar
paper_authors.type: user
paper_authors.user: amitz
paper_authors._id: 677cd67175191fdc79cbcc07
paper_authors.name: Shelly Sheynin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:17:46.692Z
paper_authors._id: 63468e6ca6f101c7f9049132
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665662111525-63468e6ca6f101c7f9049132.png
paper_authors.fullname: Shelly Sheynin
paper_authors.type: user
paper_authors.user: shellysheynin
paper_authors._id: 677cd67175191fdc79cbcc08
paper_authors.name: Yaniv Taigman
paper_authors._id: 677cd67175191fdc79cbcc09
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:17:56.992Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_authors._id: 677cd67175191fdc79cbcc0a
paper_authors.name: Sagie Benaim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:18:03.458Z
paper_authors._id: 6345a9b9a8c2ff9f1377faab
paper_authors.avatarUrl: /avatars/a5f2b999ef8b967b2af9f41afcd9d475.svg
paper_authors.fullname: Sagie Benaim
paper_authors.type: user
paper_authors.user: sagiebenaim
paper_authors._id: 677cd67175191fdc79cbcc0b
paper_authors.name: Adam Polyak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T13:18:10.411Z
paper_authors._id: 6304d514dae2eb7d08413d62
paper_authors.avatarUrl: /avatars/02a571bc791b78d3993d9a0484b70a29.svg
paper_authors.fullname: Adam Polyak
paper_authors.type: user
paper_authors.user: adampo
paper_title: Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video
    Generation
paper_summary: We consider the task of Image-to-Video (I2V) generation, which involves
  transforming static images into realistic video sequences based on a textual
  description. While recent advancements produce photorealistic outputs, they
  frequently struggle to create videos with accurate and consistent object
  motion, especially in multi-object scenarios. To address these limitations, we
  propose a two-stage compositional framework that decomposes I2V generation
  into: (i) An explicit intermediate representation generation stage, followed by
  (ii) A video generation stage that is conditioned on this representation. Our
  key innovation is the introduction of a mask-based motion trajectory as an
  intermediate representation, that captures both semantic object information and
  motion, enabling an expressive but compact representation of motion and
  semantics. To incorporate the learned representation in the second stage, we
  utilize object-level attention objectives. Specifically, we consider a spatial,
  per-object, masked-cross attention objective, integrating object-specific
  prompts into corresponding latent space regions and a masked spatio-temporal
  self-attention objective, ensuring frame-to-frame consistency for each object.
  We evaluate our method on challenging benchmarks with multi-object and
  high-motion scenarios and empirically demonstrate that the proposed method
  achieves state-of-the-art results in temporal coherence, motion realism, and
  text-prompt faithfulness. Additionally, we introduce \benchmark, a new
  challenging benchmark for single-object and multi-object I2V generation, and
  demonstrate our method's superiority on this benchmark. Project page is
  available at https://guyyariv.github.io/TTM/.
paper_ai_keywords: Image-to-Video (I2V) generation photorealistic outputs multi-object scenarios two-stage compositional framework intermediate representation generation stage video generation stage mask-based motion trajectory semantic object information motion expressive but compact representation object-level attention objectives spatial, per-object, masked-cross attention objective object-specific prompts latent space regions masked spatio-temporal self-attention objective frame-to-frame consistency temporal coherence motion realism text-prompt faithfulness \benchmark