Shepard Convolutional Neural Networks
Deep learning has recently been introduced to the field of low-level computer vision and image processing. Promising results have been obtained in a number of tasks including super-resolution, inpainting, deconvolution, filtering, etc. However, previously adopted neural network approaches such as convolutional neural networks and sparse auto-encoders are inherently with translation invariant operators. We found this property prevents the deep learning approaches from outperforming the state-of-the-art if the task itself requires translation variant interpolation (TVI). In this paper, we draw on Shepard interpolation and design Shepard Convolutional Neural Networks (ShCNN) which efficiently realizes endto-end trainable TVI operators in the network. We show that by adding only a few feature maps in the new Shepard layers, the network is able to achieve stronger results than a much deeper architecture. Superior performance on both image inpainting and super-resolution is obtained where our system outperforms previous ones while keeping the running time competitive.
PAPER SUMMARYA recent work [4] proposed using a deep CNN for super-resolution.Although it generated impressive results, it could not outperform a state-of-the-art super-resolution method based on sparse coding [10].The authors of this paper argue that this limitation is attributed to the inability of the CNN model in [4] to handle translation variant interpolation (TVI).To address this problem, they propose using an old interpolation method to develop a CNN variant called Shepard convolutional neural network (ShCNN).ShCNN is compared with other methods on inpainting and super-resolution tasks.CRITERION 1: QUALITYIt is widely perceived by many researchers that the translation-invariant property of conventional CNN, though promising for tasks such as robust object recognition, may not be very suitable for many other computer vision and image processing tasks.For example, in the context of visual tracking (not even a low-level image processing task), it was pointed out by Wang and Yeung in "Learning a deep compact image representation for visual tracking" (NIPS 2013) that a translation-variant CNN is needed for tracking applications.Likewise, it is reasonable to put forward the working hypothesis that introducing TVI to CNN would help to improve the results for inpainting and super-resolution tasks.Nevertheless, the authors have not provided justification on why Shepard interpolation is adopted here.Shepard interpolation is essentially a kernel that imposes an inverse distance weighting function.What is so special about Shepard interpolation that it is chosen here instead of many other kernel choices?Does it have computational advantages, or representational advantages, or both?Justification is missing in the paper.While it is reasonable to use masks for the inpainting task, is super-resolution with masks also a reasonable problem setting?The proposed method is evaluated empirically using a quantitative performance measure (PSNR).There is no discussion on the weaknesses of the proposed method or even analysis of some failure cases.Since ShCNN adds one or two interpolation layers on top of the original CNN, a more fair comparison is to add the same number of conventional convolutional layers.CRITERION 2: CLARITYAlthough the proposed extension to CNN is relatively simple, the paper presentation falls short of describing the proposed changes clearly and precisely.Since the paper has not reached the page limit of 9 pages, there should be plenty of space to provide more details about the model and algorithm.For example, exactly how the convolution form can be rewritten equivalently based on matrix multiplications using the unrolled matrices should beprovided in greater depth in Section 4.1.How can weight sharing be achieved using the equivalent scheme?The data preprocessing procedure in the experiments is also confusing.It seems that some pixels of the high-resolution images are kept unchanged when generating the low-resolution images and they are used to specify the masks for ShCNN.If this is the case, it may raise a fairness concern as the other methods compared (e.g., [4]) do not use any mask information.With the mask information, ShCNN somehow already knows part of the "ground truth" of the super-resolved images.Confusion is also caused by some problems with the paper writing and formatting.For example, on line #071: "... learn both translational variant operations ..." Since "both" refers to two things, what is the second thing besides "translational variant operations"?Or do you mean such operations for inpainting and super-resolution?Also, in the first paragraph on page 4, is there any difference between M and \mathbf{M}?Or is it simply a formatting error?It is very confusing to use both regular and bold fonts for matrices (e.g., I and \mathbf{K}).Moreover, sometimes it is switched between the two font styles (e.g., \mathbf{K} and K (line #240)).On the last line of page 4, what is UM?Also, what are the parameters needed for specifying the kernel \mathbf{K}?As said in the beginning of Section 5, experiments have been conducted on both inpainting and super-resolution tasks.However, the subsequent results presented seem to be for super-resolution only.To make it less cluttered, the unit "dB" can be left out in the table entries but only mentioned (once) in the figure caption.Besides, the writing has much room for improvement.Here are just a few of the many examples of language errors: #029: "outperfromed" should be written as "outperformed" #035: "In the past a few years" should be written as "In the past few years" #039: "Encouraging results has" should be written as "Encouraging results have" #044: "when applying to" should be written as "when applied to" and many more (particularly in later sections).There are also some formatting problems.For example, there should be a space before a citation, i.e., "in [8]" but not "in[8]".This problem happens a few times.CRITERION 3: ORIGINALITYShepard interpolation is an old method proposed in 1968.Since the optimization variables are in both the numerator and denominator of the fraction in Eq (1), a more sophisticated learning algorithm (as opposed to applying brute-force back-propagation) would help to handle this highly nonlinear optimization problem better.Unfortunately such an exploration is missing.In its current form, the originality of this paper appears minimal.On the problem of the previous methods discussed in the last paragraph of Section 3.1 and illustrated in Figure 2, actually it is discussed in [5] that K-SVD can already solve this problem.It should be mentioned in this paper.The difference is mainly between blind image inpainting and non-blind image inpainting.In the context of existing work, it weakens the claimed contribution of this paper.CRITERION 4: SIGNIFICANCEThe paper, especially its early parts, makes some claims that seem a bit too general.Is the proposed CNN variant for all TVI operations?Since experimental evaluation has only been done on super-resolution, it is probably more appropriate to make the claim for this particular task only.Super-resolution is always difficult to evaluate.It has been found and reported by researchers before that a method that gives higher PSNR may not give visually better result as judged by humans.One example is that if a method focuses only on the edges (primal sketch) by increasing their resolution, the super-resolved images may look better than those obtained by another method giving higher PSNR.For a new super-resolution method to bring about significant contribution, it is worth supplementing quantitative comparison with qualitative comparison involving humans.This possibility is now more feasible with the emergence of crowdsourcing platforms.