Rethinking Image Evaluation in Super-Resolution
While recent advancing image super-resolution (SR) techniques are continually
improving the perceptual quality of their outputs, they can usually fail in
quantitative evaluations. This inconsistency leads to a growing distrust in
existing image metrics for SR evaluations. Though image evaluation depends on
both the metric and the reference ground truth (GT), researchers typically do
not inspect the role of GTs, as they are generally accepted as `perfect'
references. However, due to the data being collected in the early years and the
ignorance of controlling other types of distortions, we point out that GTs in
existing SR datasets can exhibit relatively poor quality, which leads to biased
evaluations. Following this observation, in this paper, we are interested in
the following questions: Are GT images in existing SR datasets 100% trustworthy
for model evaluations? How does GT quality affect this evaluation? And how to
make fair evaluations if there exist imperfect GTs? To answer these questions,
this paper presents two main contributions. First, by systematically analyzing
seven state-of-the-art SR models across three real-world SR datasets, we show
that SR performances can be consistently affected across models by low-quality
GTs, and models can perform quite differently when GT quality is controlled.
Second, we propose a novel perceptual quality metric, Relative Quality Index
(RQI), that measures the relative quality discrepancy of image pairs, thus
issuing the biased evaluations caused by unreliable GTs. Our proposed model
achieves significantly better consistency with human opinions. We expect our
work to provide insights for the SR community on how future datasets, models,
and metrics should be developed.
67e2c563ecba0eb812dec03b
Shaolin Su
67e2c563ecba0eb812dec03c
Josep M. Rocafort
67e2c563ecba0eb812dec03d
Danna Xue
67e2c563ecba0eb812dec03e
David Serrano-Lozano
claimed_verified
2025-03-25T19:02:21.584Z
6479b418bfaa9e96b850959b
https://cdn-avatars.huggingface.co/v1/production/uploads/6479b418bfaa9e96b850959b/qQSSgCK2M8ATutL6H0Yvv.jpeg
David Serrano Lozano
user
davidserra9
67e2c563ecba0eb812dec03f
Lei Sun
67e2c563ecba0eb812dec040
Javier Vazquez-Corral
Rethinking Image Evaluation in Super-Resolution
While recent advancing image super-resolution (SR) techniques are continually
improving the perceptual quality of their outputs, they can usually fail in
quantitative evaluations. This inconsistency leads to a growing distrust in
existing image metrics for SR evaluations. Though image evaluation depends on
both the metric and the reference ground truth (GT), researchers typically do
not inspect the role of GTs, as they are generally accepted as `perfect'
references. However, due to the data being collected in the early years and the
ignorance of controlling other types of distortions, we point out that GTs in
existing SR datasets can exhibit relatively poor quality, which leads to biased
evaluations. Following this observation, in this paper, we are interested in
the following questions: Are GT images in existing SR datasets 100% trustworthy
for model evaluations? How does GT quality affect this evaluation? And how to
make fair evaluations if there exist imperfect GTs? To answer these questions,
this paper presents two main contributions. First, by systematically analyzing
seven state-of-the-art SR models across three real-world SR datasets, we show
that SR performances can be consistently affected across models by low-quality
GTs, and models can perform quite differently when GT quality is controlled.
Second, we propose a novel perceptual quality metric, Relative Quality Index
(RQI), that measures the relative quality discrepancy of image pairs, thus
issuing the biased evaluations caused by unreliable GTs. Our proposed model
achieves significantly better consistency with human opinions. We expect our
work to provide insights for the SR community on how future datasets, models,
and metrics should be developed.
The paper examines the impact of ground truth quality on image super-resolution evaluations and proposes a new perceptual quality metric to address bias caused by imperfect ground truths.
image super-resolution perceptual quality ground truth Relative Quality Index (RQI) quality discrepancy