title: Cinemo: Consistent and Controllable Image Animation with Motion
    Diffusion Models
summary: Diffusion models have achieved great progress in image animation due to
  powerful generative capabilities. However, maintaining spatio-temporal
  consistency with detailed information from the input static image over time
  (e.g., style, background, and object of the input static image) and ensuring
  smoothness in animated video narratives guided by textual prompts still remains
  challenging. In this paper, we introduce Cinemo, a novel image animation
  approach towards achieving better motion controllability, as well as stronger
  temporal consistency and smoothness. In general, we propose three effective
  strategies at the training and inference stages of Cinemo to accomplish our
  goal. At the training stage, Cinemo focuses on learning the distribution of
  motion residuals, rather than directly predicting subsequent via a motion
  diffusion model. Additionally, a structural similarity index-based strategy is
  proposed to enable Cinemo to have better controllability of motion intensity.
  At the inference stage, a noise refinement technique based on discrete cosine
  transformation is introduced to mitigate sudden motion changes. Such three
  strategies enable Cinemo to produce highly consistent, smooth, and
  motion-controllable results. Compared to previous methods, Cinemo offers
  simpler and more precise user controllability. Extensive experiments against
  several state-of-the-art methods, including both commercial tools and research
  approaches, across multiple metrics, demonstrate the effectiveness and
  superiority of our proposed approach.
paper_authors._id: 669f4076352220a858ff7018
paper_authors.name: Xin Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T13:23:25.096Z
paper_authors._id: 64a51faf9ec0167a99a4d114
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a51faf9ec0167a99a4d114/IlJS0mwkF_ZIebaQrbJIl.jpeg
paper_authors.fullname: Xin Ma
paper_authors.type: user
paper_authors.user: maxin-cn
paper_authors._id: 669f4076352220a858ff7019
paper_authors.name: Yaohui Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T13:35:16.866Z
paper_authors._id: 63201256c6b20f03c829c4b8
paper_authors.avatarUrl: /avatars/a42092119777d65e60b12eb5ba0e45f1.svg
paper_authors.fullname: Yaohui Wang
paper_authors.type: user
paper_authors.user: YaohuiW
paper_authors._id: 669f4076352220a858ff701a
paper_authors.name: Gengyu Jia
paper_authors._id: 669f4076352220a858ff701b
paper_authors.name: Xinyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T13:34:47.904Z
paper_authors._id: 643e943a70c6a27621eb1c89
paper_authors.avatarUrl: /avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
paper_authors.fullname: Xinyuan Chen
paper_authors.type: user
paper_authors.user: AriaChen
paper_authors._id: 669f4076352220a858ff701c
paper_authors.name: Yuan-Fang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T13:34:39.399Z
paper_authors._id: 6311aac5ae8896941da1b8be
paper_authors.avatarUrl: /avatars/e694dfbf2a13b2a06e940d05b6ebd545.svg
paper_authors.fullname: Yuan-Fang Li
paper_authors.type: user
paper_authors.user: yuanfang
paper_authors._id: 669f4076352220a858ff701d
paper_authors.name: Cunjian Chen
paper_authors._id: 669f4076352220a858ff701e
paper_authors.name: Yu Qiao
paper_title: Cinemo: Consistent and Controllable Image Animation with Motion
    Diffusion Models
paper_summary: Diffusion models have achieved great progress in image animation due to
  powerful generative capabilities. However, maintaining spatio-temporal
  consistency with detailed information from the input static image over time
  (e.g., style, background, and object of the input static image) and ensuring
  smoothness in animated video narratives guided by textual prompts still remains
  challenging. In this paper, we introduce Cinemo, a novel image animation
  approach towards achieving better motion controllability, as well as stronger
  temporal consistency and smoothness. In general, we propose three effective
  strategies at the training and inference stages of Cinemo to accomplish our
  goal. At the training stage, Cinemo focuses on learning the distribution of
  motion residuals, rather than directly predicting subsequent via a motion
  diffusion model. Additionally, a structural similarity index-based strategy is
  proposed to enable Cinemo to have better controllability of motion intensity.
  At the inference stage, a noise refinement technique based on discrete cosine
  transformation is introduced to mitigate sudden motion changes. Such three
  strategies enable Cinemo to produce highly consistent, smooth, and
  motion-controllable results. Compared to previous methods, Cinemo offers
  simpler and more precise user controllability. Extensive experiments against
  several state-of-the-art methods, including both commercial tools and research
  approaches, across multiple metrics, demonstrate the effectiveness and
  superiority of our proposed approach.
paper_ai_summary: Cinemo addresses spatio-temporal consistency and smoothness in image animation by learning motion residuals and applying noise refinement techniques, achieving superior performance compared to existing methods.
paper_ai_keywords: diffusion models motion residuals structural similarity index noise refinement discrete cosine transformation image animation motion controllability spatio-temporal consistency smoothness