title: Rethinking Image Evaluation in Super-Resolution
summary: While recent advancing image super-resolution (SR) techniques are continually
  improving the perceptual quality of their outputs, they can usually fail in
  quantitative evaluations. This inconsistency leads to a growing distrust in
  existing image metrics for SR evaluations. Though image evaluation depends on
  both the metric and the reference ground truth (GT), researchers typically do
  not inspect the role of GTs, as they are generally accepted as `perfect'
  references. However, due to the data being collected in the early years and the
  ignorance of controlling other types of distortions, we point out that GTs in
  existing SR datasets can exhibit relatively poor quality, which leads to biased
  evaluations. Following this observation, in this paper, we are interested in
  the following questions: Are GT images in existing SR datasets 100% trustworthy
  for model evaluations? How does GT quality affect this evaluation? And how to
  make fair evaluations if there exist imperfect GTs? To answer these questions,
  this paper presents two main contributions. First, by systematically analyzing
  seven state-of-the-art SR models across three real-world SR datasets, we show
  that SR performances can be consistently affected across models by low-quality
  GTs, and models can perform quite differently when GT quality is controlled.
  Second, we propose a novel perceptual quality metric, Relative Quality Index
  (RQI), that measures the relative quality discrepancy of image pairs, thus
  issuing the biased evaluations caused by unreliable GTs. Our proposed model
  achieves significantly better consistency with human opinions. We expect our
  work to provide insights for the SR community on how future datasets, models,
  and metrics should be developed.
paper_authors._id: 67e2c563ecba0eb812dec03b
paper_authors.name: Shaolin Su
paper_authors._id: 67e2c563ecba0eb812dec03c
paper_authors.name: Josep M. Rocafort
paper_authors._id: 67e2c563ecba0eb812dec03d
paper_authors.name: Danna Xue
paper_authors._id: 67e2c563ecba0eb812dec03e
paper_authors.name: David Serrano-Lozano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T19:02:21.584Z
paper_authors._id: 6479b418bfaa9e96b850959b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6479b418bfaa9e96b850959b/qQSSgCK2M8ATutL6H0Yvv.jpeg
paper_authors.fullname: David Serrano Lozano
paper_authors.type: user
paper_authors.user: davidserra9
paper_authors._id: 67e2c563ecba0eb812dec03f
paper_authors.name: Lei Sun
paper_authors._id: 67e2c563ecba0eb812dec040
paper_authors.name: Javier Vazquez-Corral
paper_title: Rethinking Image Evaluation in Super-Resolution
paper_summary: While recent advancing image super-resolution (SR) techniques are continually
  improving the perceptual quality of their outputs, they can usually fail in
  quantitative evaluations. This inconsistency leads to a growing distrust in
  existing image metrics for SR evaluations. Though image evaluation depends on
  both the metric and the reference ground truth (GT), researchers typically do
  not inspect the role of GTs, as they are generally accepted as `perfect'
  references. However, due to the data being collected in the early years and the
  ignorance of controlling other types of distortions, we point out that GTs in
  existing SR datasets can exhibit relatively poor quality, which leads to biased
  evaluations. Following this observation, in this paper, we are interested in
  the following questions: Are GT images in existing SR datasets 100% trustworthy
  for model evaluations? How does GT quality affect this evaluation? And how to
  make fair evaluations if there exist imperfect GTs? To answer these questions,
  this paper presents two main contributions. First, by systematically analyzing
  seven state-of-the-art SR models across three real-world SR datasets, we show
  that SR performances can be consistently affected across models by low-quality
  GTs, and models can perform quite differently when GT quality is controlled.
  Second, we propose a novel perceptual quality metric, Relative Quality Index
  (RQI), that measures the relative quality discrepancy of image pairs, thus
  issuing the biased evaluations caused by unreliable GTs. Our proposed model
  achieves significantly better consistency with human opinions. We expect our
  work to provide insights for the SR community on how future datasets, models,
  and metrics should be developed.
paper_ai_summary: The paper examines the impact of ground truth quality on image super-resolution evaluations and proposes a new perceptual quality metric to address bias caused by imperfect ground truths.
paper_ai_keywords: image super-resolution perceptual quality ground truth Relative Quality Index (RQI) quality discrepancy