Training deep learning based denoisers without ground truth data
Recently developed deep-learning-based denoisers often outperform state-of-the-art conventional denoisers, such as the BM3D. They are typically trained to minimize the mean squared error (MSE) between the output image of a deep neural network and a ground truth image. In deep learning based denoisers, it is important to use high quality noiseless ground truth data for high performance, but it is often challenging or even infeasible to obtain noiseless images in application areas such as hyperspectral remote sensing and medical imaging. In this article, we propose a method based on Stein's unbiased risk estimator (SURE) for training deep neural network denoisers only based on the use of noisy images. We demonstrate that our SURE-based method, without the use of ground truth data, is able to train deep neural network denoisers to yield performances close to those networks trained with ground truth, and to outperform the state-of-the-art denoiser BM3D. Further improvements were achieved when noisy test images were used for training of denoiser networks using our proposed SURE-based method. Code is available at https://github.com/Shakarim94/Net-SURE.
in medical imaging, ultra high resolution 3D MRI (sub-millimeter resolution) often requires several hours of acquisition time for a single, high quality volume, but reducing acquisition time leads to increased noise. In X-ray CT, image noise can be substantially reduced by increasing the radiation dose. Recent studies on deep learning based image denoisers [19,20] used CT images generated with normal doses as the ground truth so that denoising networks would be able to be trained to yield excellent performance. However, increased radiation dose leads to harmful effects in scanned subjects, while excessively high doses may saturate the CT detectors (e.g., in a similar manner to the acquisition of a photo of the sun without any filter). Thus, acquiring ground truth data with newly developed CT scanners seems challenging without compromising the subjects' safety.
Conventional denoising methods do not usually require noiseless ground truth images to perform denoising, but often require them for tuning parameters of image filters to elicit the best possible results (minimum MSE). In order to identify the optimal parameters of conventional denoisers without ground truth data, several works have been conducted [21,22] using Stein's unbiased risk estimator (SURE) [23], which is an unbiased MSE estimator. For the popular non-local means (NLM) filter [24], the analytical form of SURE was used to optimize the denoiser performance [25,26,27]. For denoisers whose analytical forms of SURE are not available, Ramani et al. [28] proposed a MonteCarlo-based SURE (MC-SURE) method to determine near-optimal denoising parameters based on the brute-force search of the parameter space. Deledalle et al. [29] investigated the approximation of a weak gradient of SURE to optimize parameters using the quasi-Newton algorithm. However, since this method requires the computation of the weak Jacobian, it is not applicable to high dimensional parameter spaces, such as deep neural networks.
We propose a SURE-based training method for deep neural network denoisers without ground truth data. In Section 2, we review key results elicited from SURE and MC-SURE. Subsequently, in Section 3, we describe our proposed method using MC-SURE and a stochastic gradient for training deep learning based image denoisers. In Section 4, simulation results are presented for (a) conventional state-of-the-art denoiser (BM3D), (b) deep learning based denoiser trained with BM3D as the ground truth, (c) the same deep neural network denoiser with the proposed SURE training without the ground truth, and (d) the same denoiser network with ground truth data as a reference. Section 5 concludes this article by discussing several potential issues for further studies.
The submission "Training deep learning denoisers without ground truth data" transfers methods from risk estimation to the setting of convolutional neural networks for denoising. The usual minimization of the l2-loss between ground truth and training data is replaced by the minimization over an unbiased estimator over training data, sampled in a Monte-Carlo fashion. The submission highlights how previous techniques for unbiased parameter estimation can be translated into the CNNs and shows very intriguing results, training without ground truth data.  A missing aspect that has to be addressed is the existence of minimizer of the SURE estimator (equation (13)) - it is easy to contruct simple (e.g. linear) networks and corresponding training data such that no minimizer \theta of (13) exists. The function value is not necessarily bounded from below, and the infimum over (13) becomes minus infinity. How can such cases be excluded, either by assumptions on the data / the number of free parameters, or by additional regularization on theta?  As minor comments  - I think it would be fair to state that the interesting idea of including test data during training also increases the inference speed significantly (e.g. to the 15 hours mentioned). - Along the lines of including the test data during training, the authors could train on as few as a single test image, yielding an interesting connection to the recent work on "Deep Image Priors" by Ulyanov et al. - Possibly related prior work on using the SURE estimator for denoising can be found, e.g., in Donoho and Johnstone, "Adapting to unknown smoothness via Wavelet Shrinkage" or Zhang and Desai, "Adaptive Denoising based on SURE risk".   - It should be noted, that although  equation (13) is an unbiased risk estimator for the MSE loss for fixed parameters theta, the minimimum over theta is not necessarily an unbiased estimation of the MSE loss of the estimator parametrized by this minimizer, see, e.g., section 1.3 in Tibsharani and Rosset's "Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?". - Please have your submission proof-read for English style and grammar issues with a particular focus on missing articles.  - I am wondering if completing the square in equation (14), ariving at a term \|y^{(j)} + \frac{\sigma^2}{\epsilon} \tilde{n}^{(j)} - h(y^{(j)}; \theta)\|^2 and one remaining linear term provides an intering point of view. What happens if one just trains with the above quadratic term? This is just a question out of curiosity and does not have to be addressed.      In summary, the work is interesting and the ideas are clearly explained. The experiments are detailed and well-formulated and show promising results. The paper is, in general, well structured and easy to read. I recommend the acceptance of this manuscript.     I appreciate the clarifications of the authors in their rebuttal. Although it is a minor technical issue (typically avoided e.g. by weight decay), I'd like to point out that the existence of minimizers is not only a question of the network architecture, but also of the training data. I agree with the other reviewers that the limitation to (fixed) Gaussian noise is severe in practice, but I'm willing to accept such a limitation as this is one of the very first papers to train denoising networks without any ground truth. Because the general idea of the paper is very interesting and the first results are promising, I keep my score and recommend the acceptance of this manuscript.