title: STAR: Spatial-Temporal Augmentation with Text-to-Video Models for
    Real-World Video Super-Resolution
summary: Image diffusion models have been adapted for real-world video
  super-resolution to tackle over-smoothing issues in GAN-based methods. However,
  these models struggle to maintain temporal consistency, as they are trained on
  static images, limiting their ability to capture temporal dynamics effectively.
  Integrating text-to-video (T2V) models into video super-resolution for improved
  temporal modeling is straightforward. However, two key challenges remain:
  artifacts introduced by complex degradations in real-world scenarios, and
  compromised fidelity due to the strong generative capacity of powerful T2V
  models (e.g., CogVideoX-5B). To enhance the spatio-temporal quality of
  restored videos, we introduce~\name
  (Spatial-Temporal Augmentation with T2V models for
  Real-world video super-resolution), a novel approach that leverages
  T2V models for real-world video super-resolution, achieving realistic spatial
  details and robust temporal consistency. Specifically, we introduce a Local
  Information Enhancement Module (LIEM) before the global attention block to
  enrich local details and mitigate degradation artifacts. Moreover, we propose a
  Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus
  on different frequency components across diffusion steps. Extensive experiments
  demonstrate~\name~outperforms state-of-the-art methods on both
  synthetic and real-world datasets.
paper_authors._id: 677c986c1b9a7499c3644ef0
paper_authors.name: Rui Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:41:59.722Z
paper_authors._id: 660a7ecf14cfe4973e0acfe1
paper_authors.avatarUrl: /avatars/e488058397f2b7a617515a4f721a9a00.svg
paper_authors.fullname: Rui Xie
paper_authors.type: user
paper_authors.user: SherryX
paper_authors._id: 677c986c1b9a7499c3644ef1
paper_authors.name: Yinhong Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T13:16:18.989Z
paper_authors._id: 6548f92f5a65d02afb4a3021
paper_authors.avatarUrl: /avatars/6bde44107ea0d1986356563f0738d2b8.svg
paper_authors.fullname: Yinhong Liu
paper_authors.type: user
paper_authors.user: Solitude-liu
paper_authors._id: 677c986c1b9a7499c3644ef2
paper_authors.name: Penghao Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T13:02:04.376Z
paper_authors._id: 667e5d480f66814a82e56420
paper_authors.avatarUrl: /avatars/33fd797c7690959fed7449666a2502bf.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: LeonZhouph
paper_authors._id: 677c986c1b9a7499c3644ef3
paper_authors.name: Chen Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:57:59.285Z
paper_authors._id: 660103ec4ae78d4ded4633fc
paper_authors.avatarUrl: /avatars/efce106d70f5d092bf44d0638aa49984.svg
paper_authors.fullname: CHEN Zhao
paper_authors.type: user
paper_authors.user: chenzhao
paper_authors._id: 677c986c1b9a7499c3644ef4
paper_authors.name: Jun Zhou
paper_authors._id: 677c986c1b9a7499c3644ef5
paper_authors.name: Kai Zhang
paper_authors._id: 677c986c1b9a7499c3644ef6
paper_authors.name: Zhenyu Zhang
paper_authors._id: 677c986c1b9a7499c3644ef7
paper_authors.name: Jian Yang
paper_authors._id: 677c986c1b9a7499c3644ef8
paper_authors.name: Zhenheng Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:58:14.804Z
paper_authors._id: 6421183b69a2c2933882d652
paper_authors.avatarUrl: /avatars/66813a8fa22915087cccd4dbfb945ca7.svg
paper_authors.fullname: Zhenheng Yang
paper_authors.type: user
paper_authors.user: zhenheny
paper_authors._id: 677c986c1b9a7499c3644ef9
paper_authors.name: Ying Tai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-07T09:58:21.644Z
paper_authors._id: 65734004769f3ee9bde1af10
paper_authors.avatarUrl: /avatars/d6310ed861972fd691687d8f47413f33.svg
paper_authors.fullname: Ying Tai
paper_authors.type: user
paper_authors.user: yingtai
paper_title: STAR: Spatial-Temporal Augmentation with Text-to-Video Models for
    Real-World Video Super-Resolution
paper_summary: Image diffusion models have been adapted for real-world video
  super-resolution to tackle over-smoothing issues in GAN-based methods. However,
  these models struggle to maintain temporal consistency, as they are trained on
  static images, limiting their ability to capture temporal dynamics effectively.
  Integrating text-to-video (T2V) models into video super-resolution for improved
  temporal modeling is straightforward. However, two key challenges remain:
  artifacts introduced by complex degradations in real-world scenarios, and
  compromised fidelity due to the strong generative capacity of powerful T2V
  models (e.g., CogVideoX-5B). To enhance the spatio-temporal quality of
  restored videos, we introduce~\name
  (Spatial-Temporal Augmentation with T2V models for
  Real-world video super-resolution), a novel approach that leverages
  T2V models for real-world video super-resolution, achieving realistic spatial
  details and robust temporal consistency. Specifically, we introduce a Local
  Information Enhancement Module (LIEM) before the global attention block to
  enrich local details and mitigate degradation artifacts. Moreover, we propose a
  Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus
  on different frequency components across diffusion steps. Extensive experiments
  demonstrate~\name~outperforms state-of-the-art methods on both
  synthetic and real-world datasets.
paper_ai_summary: A novel approach using text-to-video models with local information enhancement and dynamic frequency loss improves spatio-temporal quality in real-world video super-resolution.
paper_ai_keywords: image diffusion models T2V models video super-resolution temporal consistency Local Information Enhancement Module Dynamic Frequency Loss