Tuning-Free Noise Rectification for High Fidelity Image-to-Video
  Generation
Image-to-video (I2V) generation tasks always suffer from keeping high
fidelity in the open domains. Traditional image animation techniques primarily
focus on specific domains such as faces or human poses, making them difficult
to generalize to open domains. Several recent I2V frameworks based on diffusion
models can generate dynamic content for open domain images but fail to maintain
fidelity. We found that two main factors of low fidelity are the loss of image
details and the noise prediction biases during the denoising process. To this
end, we propose an effective method that can be applied to mainstream video
diffusion models. This method achieves high fidelity based on supplementing
more precise image information and noise rectification. Specifically, given a
specified image, our method first adds noise to the input image latent to keep
more details, then denoises the noisy latent with proper rectification to
alleviate the noise prediction biases. Our method is tuning-free and
plug-and-play. The experimental results demonstrate the effectiveness of our
approach in improving the fidelity of generated videos. For more image-to-video
generated results, please refer to the project website:
https://noise-rectification.github.io.
65e7d0103fb61ba9c610aedc
Weijie Li
65e7d0103fb61ba9c610aedd
Litong Gong
admin_assigned
2024-03-06T10:40:55.980Z
6322bc68a97afe3c4c6dd8a7
/avatars/c014508235b57a9750621755efc189c6.svg
Litong.G
user
culeao
65e7d0103fb61ba9c610aede
Yiran Zhu
admin_assigned
2024-03-06T11:38:37.782Z
63ecce0054bba678151814cf
https://cdn-avatars.huggingface.co/v1/production/uploads/1676463764139-63ecce0054bba678151814cf.png
y1Ran
user
y1Ran
65e7d0103fb61ba9c610aedf
Fanda Fan
admin_assigned
2024-03-06T10:41:31.788Z
650152bed522560505e0e092
/avatars/9fb3eee1b23859bfae6ca7d3c15e4bff.svg
Fanda Fan
user
stevenfan
65e7d0103fb61ba9c610aee0
Biao Wang
claimed_verified
2024-03-06T15:34:27.063Z
64dcd9de0a0efd28e3b846e3
/avatars/0732be326b1d3658118f669dd283d582.svg
ictwangbiao
user
ictwangbiao
65e7d0103fb61ba9c610aee1
Tiezheng Ge
65e7d0103fb61ba9c610aee2
Bo Zheng
Tuning-Free Noise Rectification for High Fidelity Image-to-Video
  Generation
Image-to-video (I2V) generation tasks always suffer from keeping high
fidelity in the open domains. Traditional image animation techniques primarily
focus on specific domains such as faces or human poses, making them difficult
to generalize to open domains. Several recent I2V frameworks based on diffusion
models can generate dynamic content for open domain images but fail to maintain
fidelity. We found that two main factors of low fidelity are the loss of image
details and the noise prediction biases during the denoising process. To this
end, we propose an effective method that can be applied to mainstream video
diffusion models. This method achieves high fidelity based on supplementing
more precise image information and noise rectification. Specifically, given a
specified image, our method first adds noise to the input image latent to keep
more details, then denoises the noisy latent with proper rectification to
alleviate the noise prediction biases. Our method is tuning-free and
plug-and-play. The experimental results demonstrate the effectiveness of our
approach in improving the fidelity of generated videos. For more image-to-video
generated results, please refer to the project website:
https://noise-rectification.github.io.
The proposed method enhances the fidelity of image-to-video generation by adding noise to image latent information and rectifying noise during the denoising process in diffusion models.
diffusion models denoising process noise prediction biases noise rectification