title: Motion-I2V: Consistent and Controllable Image-to-Video Generation with
    Explicit Motion Modeling
summary: We introduce Motion-I2V, a novel framework for consistent and controllable
  image-to-video generation (I2V). In contrast to previous methods that directly
  learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into
  two stages with explicit motion modeling. For the first stage, we propose a
  diffusion-based motion field predictor, which focuses on deducing the
  trajectories of the reference image's pixels. For the second stage, we propose
  motion-augmented temporal attention to enhance the limited 1-D temporal
  attention in video latent diffusion models. This module can effectively
  propagate reference image's feature to synthesized frames with the guidance of
  predicted trajectories from the first stage. Compared with existing methods,
  Motion-I2V can generate more consistent videos even at the presence of large
  motion and viewpoint variation. By training a sparse trajectory ControlNet for
  the first stage, Motion-I2V can support users to precisely control motion
  trajectories and motion regions with sparse trajectory and region annotations.
  This offers more controllability of the I2V process than solely relying on
  textual instructions. Additionally, Motion-I2V's second stage naturally
  supports zero-shot video-to-video translation. Both qualitative and
  quantitative comparisons demonstrate the advantages of Motion-I2V over prior
  approaches in consistent and controllable image-to-video generation.
paper_authors._id: 65b8611d6169e07d8d7bc979
paper_authors.name: Xiaoyu Shi
paper_authors._id: 65b8611d6169e07d8d7bc97a
paper_authors.name: Zhaoyang Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:32:52.442Z
paper_authors._id: 64c26e0e400274d129a10d5d
paper_authors.avatarUrl: /avatars/c1debf3d7f61c2ea01d9a25d7e332712.svg
paper_authors.fullname: Zhaoyang Huang
paper_authors.type: user
paper_authors.user: DrinkingCoder
paper_authors._id: 65b8611d6169e07d8d7bc97b
paper_authors.name: Fu-Yun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T08:06:41.271Z
paper_authors._id: 63e9e92f20c109718713f5eb
paper_authors.avatarUrl: /avatars/9ff312e854d803e1a2e9e685a21d12f8.svg
paper_authors.fullname: Fu-Yun Wang
paper_authors.type: user
paper_authors.user: wangfuyun
paper_authors._id: 65b8611d6169e07d8d7bc97c
paper_authors.name: Weikang Bian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T09:00:01.707Z
paper_authors._id: 65b8b840a81930c593073370
paper_authors.avatarUrl: /avatars/31b399d3da33e8c02f9e06a16dd85779.svg
paper_authors.fullname: Weikang BIAN
paper_authors.type: user
paper_authors.user: wkbian
paper_authors._id: 65b8611d6169e07d8d7bc97d
paper_authors.name: Dasong Li
paper_authors._id: 65b8611d6169e07d8d7bc97e
paper_authors.name: Yi Zhang
paper_authors._id: 65b8611d6169e07d8d7bc97f
paper_authors.name: Manyuan Zhang
paper_authors._id: 65b8611d6169e07d8d7bc980
paper_authors.name: Ka Chun Cheung
paper_authors._id: 65b8611d6169e07d8d7bc981
paper_authors.name: Simon See
paper_authors._id: 65b8611d6169e07d8d7bc982
paper_authors.name: Hongwei Qin
paper_authors._id: 65b8611d6169e07d8d7bc983
paper_authors.name: Jifeng Da
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:37:55.235Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 65b8611d6169e07d8d7bc984
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:49:23.281Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: Motion-I2V: Consistent and Controllable Image-to-Video Generation with
    Explicit Motion Modeling
paper_summary: We introduce Motion-I2V, a novel framework for consistent and controllable
  image-to-video generation (I2V). In contrast to previous methods that directly
  learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into
  two stages with explicit motion modeling. For the first stage, we propose a
  diffusion-based motion field predictor, which focuses on deducing the
  trajectories of the reference image's pixels. For the second stage, we propose
  motion-augmented temporal attention to enhance the limited 1-D temporal
  attention in video latent diffusion models. This module can effectively
  propagate reference image's feature to synthesized frames with the guidance of
  predicted trajectories from the first stage. Compared with existing methods,
  Motion-I2V can generate more consistent videos even at the presence of large
  motion and viewpoint variation. By training a sparse trajectory ControlNet for
  the first stage, Motion-I2V can support users to precisely control motion
  trajectories and motion regions with sparse trajectory and region annotations.
  This offers more controllability of the I2V process than solely relying on
  textual instructions. Additionally, Motion-I2V's second stage naturally
  supports zero-shot video-to-video translation. Both qualitative and
  quantitative comparisons demonstrate the advantages of Motion-I2V over prior
  approaches in consistent and controllable image-to-video generation.
paper_ai_keywords: Motion-I2V image-to-video generation (I2V) diffusion-based motion field predictor trajectories pixel trajectories temporal attention motion-augmented temporal attention video latent diffusion models synthesized frames sparse trajectory ControlNet motion trajectories motion regions sparse trajectory and region unifications zero-shot video-to-video translation consistent image-to-video generation controllable image-to-video generation Motion-I2V diffusion-based motion field predictor trajectories pixel trajectories temporal attention motion-augmented temporal attention video latent diffusion models synthesized frames sparse trajectory ControlNet motion trajectories motion regions sparse trajectory and region unifications zero-shot video-to-video translation consistent image-to-video generation controllable image-to-video generation