Cinemo: Consistent and Controllable Image Animation with Motion
  Diffusion Models
Diffusion models have achieved great progress in image animation due to
powerful generative capabilities. However, maintaining spatio-temporal
consistency with detailed information from the input static image over time
(e.g., style, background, and object of the input static image) and ensuring
smoothness in animated video narratives guided by textual prompts still remains
challenging. In this paper, we introduce Cinemo, a novel image animation
approach towards achieving better motion controllability, as well as stronger
temporal consistency and smoothness. In general, we propose three effective
strategies at the training and inference stages of Cinemo to accomplish our
goal. At the training stage, Cinemo focuses on learning the distribution of
motion residuals, rather than directly predicting subsequent via a motion
diffusion model. Additionally, a structural similarity index-based strategy is
proposed to enable Cinemo to have better controllability of motion intensity.
At the inference stage, a noise refinement technique based on discrete cosine
transformation is introduced to mitigate sudden motion changes. Such three
strategies enable Cinemo to produce highly consistent, smooth, and
motion-controllable results. Compared to previous methods, Cinemo offers
simpler and more precise user controllability. Extensive experiments against
several state-of-the-art methods, including both commercial tools and research
approaches, across multiple metrics, demonstrate the effectiveness and
superiority of our proposed approach.
669f4076352220a858ff7018
Xin Ma
admin_assigned
2024-07-23T13:23:25.096Z
64a51faf9ec0167a99a4d114
https://cdn-avatars.huggingface.co/v1/production/uploads/64a51faf9ec0167a99a4d114/IlJS0mwkF_ZIebaQrbJIl.jpeg
Xin Ma
user
maxin-cn
669f4076352220a858ff7019
Yaohui Wang
admin_assigned
2024-07-23T13:35:16.866Z
63201256c6b20f03c829c4b8
/avatars/a42092119777d65e60b12eb5ba0e45f1.svg
Yaohui Wang
user
YaohuiW
669f4076352220a858ff701a
Gengyu Jia
669f4076352220a858ff701b
Xinyuan Chen
admin_assigned
2024-07-23T13:34:47.904Z
643e943a70c6a27621eb1c89
/avatars/73ec521ab5ba84cc7908c52c0acef6ef.svg
Xinyuan Chen
user
AriaChen
669f4076352220a858ff701c
Yuan-Fang Li
admin_assigned
2024-07-23T13:34:39.399Z
6311aac5ae8896941da1b8be
/avatars/e694dfbf2a13b2a06e940d05b6ebd545.svg
Yuan-Fang Li
user
yuanfang
669f4076352220a858ff701d
Cunjian Chen
669f4076352220a858ff701e
Yu Qiao
Cinemo: Consistent and Controllable Image Animation with Motion
  Diffusion Models
Diffusion models have achieved great progress in image animation due to
powerful generative capabilities. However, maintaining spatio-temporal
consistency with detailed information from the input static image over time
(e.g., style, background, and object of the input static image) and ensuring
smoothness in animated video narratives guided by textual prompts still remains
challenging. In this paper, we introduce Cinemo, a novel image animation
approach towards achieving better motion controllability, as well as stronger
temporal consistency and smoothness. In general, we propose three effective
strategies at the training and inference stages of Cinemo to accomplish our
goal. At the training stage, Cinemo focuses on learning the distribution of
motion residuals, rather than directly predicting subsequent via a motion
diffusion model. Additionally, a structural similarity index-based strategy is
proposed to enable Cinemo to have better controllability of motion intensity.
At the inference stage, a noise refinement technique based on discrete cosine
transformation is introduced to mitigate sudden motion changes. Such three
strategies enable Cinemo to produce highly consistent, smooth, and
motion-controllable results. Compared to previous methods, Cinemo offers
simpler and more precise user controllability. Extensive experiments against
several state-of-the-art methods, including both commercial tools and research
approaches, across multiple metrics, demonstrate the effectiveness and
superiority of our proposed approach.
Cinemo addresses spatio-temporal consistency and smoothness in image animation by learning motion residuals and applying noise refinement techniques, achieving superior performance compared to existing methods.
diffusion models motion residuals structural similarity index noise refinement discrete cosine transformation image animation motion controllability spatio-temporal consistency smoothness