Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
In this paper, we propose a very deep fully convolutional encoding-decoding framework for image restoration such as denoising and super-resolution. The network is composed of multiple layers of convolution and deconvolution operators, learning end-to-end mappings from corrupted images to the original ones. The convolutional layers act as the feature extractor, which capture the abstraction of image contents while eliminating noises/corruptions. Deconvolutional layers are then used to recover the image details. We propose to symmetrically link convolutional and deconvolutional layers with skip-layer connections, with which the training converges much faster and attains a higher-quality local optimum. First, the skip connections allow the signal to be back-propagated to bottom layers directly, and thus tackles the problem of gradient vanishing, making training deep networks easier and achieving restoration performance gains consequently. Second, these skip connections pass image details from convolutional layers to deconvolutional layers, which is beneficial in recovering the original image. Significantly, with the large capacity, we can handle different levels of noises using a single model. Experimental results show that our network achieves better performance than recent state-of-the-art methods.
This paper proposed a deep fully convolutional neural network for image restoration such as denoising and super-resolution. The network is composed of multiple layers of convolution and de-convolution operators, learning end-to-end mappings from corrupted images to the ground-truth. In addition, symmetrical links are used to connect the corresponding convolution and de-convolution layers. Experiments results show that the proposed method outperforms the state-of-the-arts on image denoising and super-resolution.
The proposed method is very related to Kim et al.’s work, but the later one is not mentioned at all. At the time of submission, the arXiv version of Kim’s paper was already online, which should be cited and discussed.  Kim, Jiwon, Jung Kwon Lee, and Kyoung Mu Lee. "Accurate image super-resolution using very deep convolutional networks." arXiv preprint arXiv:1511.04587 (2015). Accepted at CVPR 2016.    In Kim et al.’s work, the model is with multiple flat convolution layers. The input is connected with the output to form a residual learning. Compared with Kim’s model, half of convolutional layers are replaced with de-convolution layers in the proposed model and it has more skip connections between the layers. It seems that the overall idea is very similar to Kim’s paper that using very deep fully convolutional layers and residual learning for image restoration. Overall, there is some advancement made by the paper, but may be not significantly enough for a NIPS paper.   1. It is claimed that this paper different corruptions using a single model. While it provides some contribution in denoising, the strategy of mixed training set has already used in Kim’s paper for different levels of super-resolution. Therefore, the contribution is not that significant.   2. In section 2.2, to validate the benefits of skip connections, it is not very convincing by comparing residual networks [11] since [11] was proposed for recognition. A more proper choice could be the version of removing the connections in the middle, with only one input-to-end connection. It is like the residual learning in Kim’s work. This experiment can validate the necessity of the middle connections.   3. I think it is weird for a paper on image restoration without any visual comparisons in the paper.  although these results are included in the supplementary material. For some results, the improvements are not very clear. Highlighting some zoomed-in regions would be helpful.    Minor:  1. Tables 1,2,3 and 6,7 have the same captions, but actually they are trained with different strategies. The captions should be changed to make such difference clear without reading the text in the paper.  2. In Fig. 3(a), why the PSNR decreased when the number of epochs increased? 3. Another work using symmetric fully convolution and de-convolution layers may be related to this paper.   Edgar Simo-Serra et al., Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup. SIGGRAPH 2016.   __________________________________________________________________ Update:  After reading the authors’ rebuttal, I still have several concerns about this paper.   The authors said that the difference between this paper and Kim’s paper is that they used more skip connections instead of a single one from the input to the output. To me, the results are not necessarily getting better by doing this. In the paper, there is no comparison between a single connection and multiple connections. Therefore, it is hard to evaluate the benefit of this. The authors only compared the multiple connections with residual learning [11], which was a different structure with a skip connection in each small block.   Another import issue of this paper is that Kim’s work is never mentioned. It could mislead the readers: it seems that all the ideas of this paper (e.g., skip connection, very deep structure, handle different noise/scale by mixing the training data) are original, but actually very similar to Kim’s work. The arXiv version of Kim’s paper was online in 2015. I am not asking the authors to compare with an arXiv paper, but the difference should be discussed since the ideas are so similar.   The results are also not very convincing.  In the supplementary results, it is hardly to see any visual improvements over existing methods. While the “numbers” (e.g. PSNR) are slighted improved by 0.1-0.2dB, it does not mean a necessary improvement. For example, in Johnson’s paper, it shows a lower PSNR gives perceptually better looking result. I agree with the authors that this paper has slightly higher PSNR than Kim's(around 0.15dB) on Set5 and BSD 100, but lower by similar amounts on Set14.   Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. "Perceptual losses for real-time style transfer and super-resolution." arXiv preprint arXiv:1603.08155 (ECCV) 2016.  Overall, I feel the major issue of this paper is the lack of proper discussions and evaluations.