DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models
<|reference_start|>DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models: This paper introduces a method for zero-shot video restoration using pre-trained image restoration diffusion models. Traditional video restoration methods often need retraining for different settings and struggle with limited generalization across various degradation types and datasets. Our approach uses a hierarchical token merging strategy for keyframes and local frames, combined with a hybrid correspondence mechanism that blends optical flow and feature-based nearest neighbor matching (latent merging). We show that our method not only achieves top performance in zero-shot video restoration but also significantly surpasses trained models in generalization across diverse datasets and extreme degradations (8$\times$ super-resolution and high-standard deviation video denoising). We present evidence through quantitative metrics and visual comparisons on various challenging datasets. Additionally, our technique works with any 2D restoration diffusion model, offering a versatile and powerful tool for video enhancement tasks without extensive retraining. This research leads to more efficient and widely applicable video restoration technologies, supporting advancements in fields that require high-quality video output. See our project page for video results and source code at https://jimmycv07.github.io/DiffIR2VR_web/.<|reference_end|>
@article{yeh2024diffir2vr-zero:,
    title={DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image
  Restoration Models},
    author={Chang-Han Yeh, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Ting-Hsuan
  Chen, Hau-Shiang Shiu, Yu-Lun Liu},
    journal={arXiv preprint arXiv:2407.01519},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.01519},
    primaryClass={cs.CV}
}