DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image
  Restoration Models
This paper introduces a method for zero-shot video restoration using
pre-trained image restoration diffusion models. Traditional video restoration
methods often need retraining for different settings and struggle with limited
generalization across various degradation types and datasets. Our approach uses
a hierarchical token merging strategy for keyframes and local frames, combined
with a hybrid correspondence mechanism that blends optical flow and
feature-based nearest neighbor matching (latent merging). We show that our
method not only achieves top performance in zero-shot video restoration but
also significantly surpasses trained models in generalization across diverse
datasets and extreme degradations (8times super-resolution and high-standard
deviation video denoising). We present evidence through quantitative metrics
and visual comparisons on various challenging datasets. Additionally, our
technique works with any 2D restoration diffusion model, offering a versatile
and powerful tool for video enhancement tasks without extensive retraining.
This research leads to more efficient and widely applicable video restoration
technologies, supporting advancements in fields that require high-quality video
output. See our project page for video results at
https://jimmycv07.github.io/DiffIR2VR_web/.
66836cfa48edefb453ab296c
Chang-Han Yeh
claimed_verified
2024-07-02T07:38:46.600Z
65db54a31ed8dc6d82837094
/avatars/668ad7105f6e3c71bbe3ff5c75e2860e.svg
Yeh, Chang-Han
user
jimmycv07
66836cfa48edefb453ab296d
Chin-Yang Lin
66836cfa48edefb453ab296e
Zhixiang Wang
claimed_verified
2025-04-08T14:25:03.814Z
6307a98795b2ab342fec0cf7
/avatars/85b261bcdda4717a6e40491f6c7b7a89.svg
Zhixiang Wang
user
wangzx1994
66836cfa48edefb453ab296f
Chi-Wei Hsiao
admin_assigned
2024-07-03T15:11:15.734Z
66600a21e5122edc2f4d1f09
/avatars/a6446ac21eada2014c59ac38ec019862.svg
Chi-Wei Hsiao
user
chiweihsiao
66836cfa48edefb453ab2970
Ting-Hsuan Chen
claimed_verified
2024-07-02T07:38:48.611Z
65d78eb72a17b13e8fa32aa6
/avatars/1c7d598d67dc0e233d65dac6d9260e1c.svg
Ting-Hsuan Chen
user
Koi953215
66836cfa48edefb453ab2971
Yu-Lun Liu
admin_assigned
2024-07-03T15:11:08.323Z
6459d5da3b6fafd9664807ab
/avatars/57430d1bbde3a2fe5586e5fbcafb0e74.svg
Yu-Lun Liu
user
yulunliu
DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image
  Restoration Models
This paper introduces a method for zero-shot video restoration using
pre-trained image restoration diffusion models. Traditional video restoration
methods often need retraining for different settings and struggle with limited
generalization across various degradation types and datasets. Our approach uses
a hierarchical token merging strategy for keyframes and local frames, combined
with a hybrid correspondence mechanism that blends optical flow and
feature-based nearest neighbor matching (latent merging). We show that our
method not only achieves top performance in zero-shot video restoration but
also significantly surpasses trained models in generalization across diverse
datasets and extreme degradations (8times super-resolution and high-standard
deviation video denoising). We present evidence through quantitative metrics
and visual comparisons on various challenging datasets. Additionally, our
technique works with any 2D restoration diffusion model, offering a versatile
and powerful tool for video enhancement tasks without extensive retraining.
This research leads to more efficient and widely applicable video restoration
technologies, supporting advancements in fields that require high-quality video
output. See our project page for video results at
https://jimmycv07.github.io/DiffIR2VR_web/.
A zero-shot video restoration method using pre-trained image restoration diffusion models with hierarchical token merging and a hybrid correspondence mechanism achieves top performance and generalization across various datasets and degradations.
zero-shot video restoration diffusion models hierarchical token merging keyframes local frames hybrid correspondence mechanism optical flow feature-based nearest neighbor matching latent merging super-resolution video denoising 2D restoration diffusion model video enhancement