title: MOFA-Video: Controllable Image Animation via Generative Motion Field
    Adaptions in Frozen Image-to-Video Diffusion Model
summary: We present MOFA-Video, an advanced controllable image animation method that
  generates video from the given image using various additional controllable
  signals (such as human landmarks reference, manual trajectories, and another
  even provided video) or their combinations. This is different from previous
  methods which only can work on a specific motion domain or show weak control
  abilities with diffusion prior. To achieve our goal, we design several
  domain-aware motion field adapters (\ie, MOFA-Adapters) to control the
  generated motions in the video generation pipeline. For MOFA-Adapters, we
  consider the temporal motion consistency of the video and generate the dense
  motion flow from the given sparse control conditions first, and then, the
  multi-scale features of the given image are wrapped as a guided feature for
  stable video diffusion generation. We naively train two motion adapters for the
  manual trajectories and the human landmarks individually since they both
  contain sparse information about the control. After training, the MOFA-Adapters
  in different domains can also work together for more controllable video
  generation.
paper_authors._id: 6659461d9ade6238ae865768
paper_authors.name: Muyao Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:04:39.074Z
paper_authors._id: 66597e53085eb651879aa4a1
paper_authors.avatarUrl: /avatars/116aa003266bb19e8e03eb7ac5633b03.svg
paper_authors.fullname: Muyao Niu
paper_authors.type: user
paper_authors.user: MyNiuuu
paper_authors._id: 6659461d9ade6238ae865769
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:04:44.551Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 6659461d9ade6238ae86576a
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:04:50.430Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 6659461d9ade6238ae86576b
paper_authors.name: Yong Zhang
paper_authors._id: 6659461d9ade6238ae86576c
paper_authors.name: Ying Shan
paper_authors._id: 6659461d9ade6238ae86576d
paper_authors.name: Yinqiang Zheng
paper_title: MOFA-Video: Controllable Image Animation via Generative Motion Field
    Adaptions in Frozen Image-to-Video Diffusion Model
paper_summary: We present MOFA-Video, an advanced controllable image animation method that
  generates video from the given image using various additional controllable
  signals (such as human landmarks reference, manual trajectories, and another
  even provided video) or their combinations. This is different from previous
  methods which only can work on a specific motion domain or show weak control
  abilities with diffusion prior. To achieve our goal, we design several
  domain-aware motion field adapters (\ie, MOFA-Adapters) to control the
  generated motions in the video generation pipeline. For MOFA-Adapters, we
  consider the temporal motion consistency of the video and generate the dense
  motion flow from the given sparse control conditions first, and then, the
  multi-scale features of the given image are wrapped as a guided feature for
  stable video diffusion generation. We naively train two motion adapters for the
  manual trajectories and the human landmarks individually since they both
  contain sparse information about the control. After training, the MOFA-Adapters
  in different domains can also work together for more controllable video
  generation.
paper_ai_keywords: controllable image animation image animation video generation human landmarks manual trajectories domain-aware motion field adapters MOFA-Adapters motion generation dense motion flow sparse control conditions multi-scale features guided feature stable video diffusion generation motion adapters