title: DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image
    Restoration Models
summary: This paper introduces a method for zero-shot video restoration using
  pre-trained image restoration diffusion models. Traditional video restoration
  methods often need retraining for different settings and struggle with limited
  generalization across various degradation types and datasets. Our approach uses
  a hierarchical token merging strategy for keyframes and local frames, combined
  with a hybrid correspondence mechanism that blends optical flow and
  feature-based nearest neighbor matching (latent merging). We show that our
  method not only achieves top performance in zero-shot video restoration but
  also significantly surpasses trained models in generalization across diverse
  datasets and extreme degradations (8times super-resolution and high-standard
  deviation video denoising). We present evidence through quantitative metrics
  and visual comparisons on various challenging datasets. Additionally, our
  technique works with any 2D restoration diffusion model, offering a versatile
  and powerful tool for video enhancement tasks without extensive retraining.
  This research leads to more efficient and widely applicable video restoration
  technologies, supporting advancements in fields that require high-quality video
  output. See our project page for video results at
  https://jimmycv07.github.io/DiffIR2VR_web/.
paper_authors._id: 66836cfa48edefb453ab296c
paper_authors.name: Chang-Han Yeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:46.600Z
paper_authors._id: 65db54a31ed8dc6d82837094
paper_authors.avatarUrl: /avatars/668ad7105f6e3c71bbe3ff5c75e2860e.svg
paper_authors.fullname: Yeh, Chang-Han
paper_authors.type: user
paper_authors.user: jimmycv07
paper_authors._id: 66836cfa48edefb453ab296d
paper_authors.name: Chin-Yang Lin
paper_authors._id: 66836cfa48edefb453ab296e
paper_authors.name: Zhixiang Wang
paper_authors._id: 66836cfa48edefb453ab296f
paper_authors.name: Chi-Wei Hsiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:11:15.734Z
paper_authors._id: 66600a21e5122edc2f4d1f09
paper_authors.avatarUrl: /avatars/a6446ac21eada2014c59ac38ec019862.svg
paper_authors.fullname: Chi-Wei Hsiao
paper_authors.type: user
paper_authors.user: chiweihsiao
paper_authors._id: 66836cfa48edefb453ab2970
paper_authors.name: Ting-Hsuan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-02T07:38:48.611Z
paper_authors._id: 65d78eb72a17b13e8fa32aa6
paper_authors.avatarUrl: /avatars/1c7d598d67dc0e233d65dac6d9260e1c.svg
paper_authors.fullname: Ting-Hsuan Chen
paper_authors.type: user
paper_authors.user: Koi953215
paper_authors._id: 66836cfa48edefb453ab2971
paper_authors.name: Yu-Lun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-03T15:11:08.323Z
paper_authors._id: 6459d5da3b6fafd9664807ab
paper_authors.avatarUrl: /avatars/57430d1bbde3a2fe5586e5fbcafb0e74.svg
paper_authors.fullname: Yu-Lun Liu
paper_authors.type: user
paper_authors.user: yulunliu
paper_title: DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image
    Restoration Models
paper_summary: This paper introduces a method for zero-shot video restoration using
  pre-trained image restoration diffusion models. Traditional video restoration
  methods often need retraining for different settings and struggle with limited
  generalization across various degradation types and datasets. Our approach uses
  a hierarchical token merging strategy for keyframes and local frames, combined
  with a hybrid correspondence mechanism that blends optical flow and
  feature-based nearest neighbor matching (latent merging). We show that our
  method not only achieves top performance in zero-shot video restoration but
  also significantly surpasses trained models in generalization across diverse
  datasets and extreme degradations (8times super-resolution and high-standard
  deviation video denoising). We present evidence through quantitative metrics
  and visual comparisons on various challenging datasets. Additionally, our
  technique works with any 2D restoration diffusion model, offering a versatile
  and powerful tool for video enhancement tasks without extensive retraining.
  This research leads to more efficient and widely applicable video restoration
  technologies, supporting advancements in fields that require high-quality video
  output. See our project page for video results at
  https://jimmycv07.github.io/DiffIR2VR_web/.
paper_ai_keywords: diffusion models zero-shot video restoration hierarchical token merging keyframes local frames hybrid correspondence mechanism optical flow feature-based nearest matching latent merging generalization datasets extreme degradations super-resolution video denoising quantitative metrics visual comparisons 2D restoration diffusion model video enhancement tasks highVA video output