Deep Mean-Shift Priors for Image Restoration
In this paper we introduce a natural image prior that directly represents a Gaussiansmoothed version of the natural image distribution. We include our prior in a formulation of image restoration as a Bayes estimator that also allows us to solve noise-blind image restoration problems. We show that the gradient of our prior corresponds to the mean-shift vector on the natural image distribution. In addition, we learn the mean-shift vector field using denoising autoencoders, and use it in a gradient descent approach to perform Bayes risk minimization. We demonstrate competitive results for noise-blind deblurring, super-resolution, and demosaicing.
Summary
-------
The paper proposes a way to do Bayesian estimation for image restoration problems by using gradient descent to minimize a bound of the Bayes risk. In particular, the paper uses Gaussian smoothing in the utility function and the image prior, which allows to establish a connection between the gradient of the log-prior and denoising auto-encoders (DAEs), which are used to learn the prior from training data. Several image restoration experiments demonstrate results on par with the state-of-the-art.
Main comments
-------------
The positive aspects of the paper are its good results (on par with state-of-the-art methods) for several image restoration tasks and the ability of the same learned prior to be used for several tasks (mostly) without re-training. Another strong aspect is the ability to jointly do noise- and kernel-blind deconvolution, which other methods typically cannot do.
I found it somewhat difficult to judge the novelty of the paper, since it seems to be heavily influenced/based on the papers by Jin et al. [14] (noise-adaptive approach, general derivation approach) and Bigdeli and Zwicker [4] (using DAEs as priors, mean-shift connection). Please properly attribute where things have been taken from previous work and make the differences to the proposed approach more clear. In particular:
- The mean-shift connection is not really discussed, just written that it is "well-known". Please explain this in more detail, or just remove it from the paper (especially from the title).
- Using a Gaussian function as/in the utility function seems to be taken from [14] (?), but not acknowledged. The entire (?) noise-adaptive approach seems to be adopted from [14], please acknowledge/distinguish more thoroughly (e.g. in lines 168-171).
Although the experiments demonstrate that the proposed method can lead to good results, many approximations and bounds are used to make it work. Can you give an explanation why the method works so well despite this?
Related to this, if I understand correctly, the whole noise sampling approach in section 3.3 is only used to make up for the fact that the learned DAE is overfitted to noisy images? I don't understand how this fixes the problem.
Regarding generalization, why is it necessary to train a different DAE for the demosaicing experiments? Shouldn't the image prior be generic?
Related to this, the noise- and kernel-blind experiments (section 5.2) use different gradient descent parameters. How sensitive is the method towards these hyper-parameters?
The paper doesn't mention at all if the method has reasonable runtimes for typical image sizes. Please provide some ballpark test runtimes for a single gradient descent step (for the various restoration tasks).
Miscellaneous
-------------
- I found the abstract hard to understand without having read the rest of the paper.
- It does seems somewhat artificial to include p'(x)/p(\tilde{x}) in Eq. 5 instead of just removing this term from the utility function and simply replacing p(x) with the smoothed prior p'(x).
- Typo in line 182: "form" -> "from"
- Please define the Gaussian function g_\sigma in Eq. 6 and later.
- Table 2:
	- EPLL/TV-L2 + NA not explained, how can they use the noise-adaptive approach?
	- Misleading results: e.g. RTF-6 [28] only trained for \sigma=2.55, doesn't make sense to evaluate for other noise levels. Please just leave the table entry blank.