Joint Sub-bands Learning with Clique Structures for Wavelet Domain Super-Resolution
Convolutional neural networks (CNNs) have recently achieved great success in single-image super-resolution (SISR). However, these methods tend to produce over-smoothed outputs and miss some textural details. To solve these problems, we propose the Super-Resolution CliqueNet (SRCliqueNet) to reconstruct the high resolution (HR) image with better textural details in the wavelet domain. The proposed SRCliqueNet firstly extracts a set of feature maps from the low resolution (LR) image by the clique blocks group. Then we send the set of feature maps to the clique up-sampling module to reconstruct the HR image. The clique upsampling module consists of four sub-nets which predict the high resolution wavelet coefficients of four sub-bands. Since we consider the edge feature properties of four sub-bands, the four sub-nets are connected to the others so that they can learn the coefficients of four sub-bands jointly. Finally we apply inverse discrete wavelet transform (IDWT) to the output of four sub-nets at the end of the clique up-sampling module to increase the resolution and reconstruct the HR image. Extensive quantitative and qualitative experiments on benchmark datasets show that our method achieves superior performance over the state-of-the-art methods.
Summary:  This paper proposes a CNN architecture called SRCliqueNet for single-image super-resolution (SISR), and it consists of two key parts, feature embedding net (FEN) and the image reconstruction net (IRN).   FEN consists of two convolutional layers and a clique block group. The first convolutional layer tries to increase the number of channels of input, which can be added with the output of the clique block group via the skip connection. The second convlutional layer tries to change the number of channels so that they can fit the input of clique block group. The clique block group concatenates features from a sequence of clique blocks, each of which has two stages: 1) the first stage does the same things as dense block, while the second stage distills the feature further. Following the idea of resnet, a clique block contains more skip connections compared with a dense block, so the information among layers can be more easily propagated.  IRN consists of a clique up-sampling module and a convolutional layer used to reduce the number of feature maps to reconstruct the HR image with RGB channels. The clique up-sampling module contains four sub networks, representing four sub-bands, LL, LH, HL and HH, in the wavelet domain, respectively. Each sub network has 1) subband extraction, 2) residual block, 3) a refinement stage that uses the high frequency blocks to refine/learn the low frequency blocks, 4) IDWT, and 5) a convolutional layer, which is used to reduce the number of channels and get the HR image.  Experiment evaluations on benchmark datasets show that that SRCliqueNet performs better than the state-of-the-art SR algorithms in terms of quantitative metrics.  Strengths:  - A novel idea of modeling the relationship among the four sub-bands in the clique up-sampling module in IRN. - This paper is technically sound with extensive quantitative evaluation of state-of-the-art SR algorithms on benchmark dataset, and experiment and visualization to demonstrate the usefulness of FEN and IRN. - This paper is well write with clear description of how SRCliqueNet works.  Weaknesses:  Overall it is a great paper. It will be good if training and testing time is also reported. This is helpful to know if it can be used in real-time SR.  Rebuttal Response:  After reading the review, I am more clear on the testing time of this approach. For now, it appears that it may not be fast enough to process real time video frame (1 second / image now based on rebuttal), but it does not decrease the value of this work to use DL on wavelet domain. I will remain my score to be 8.