Deconvolutional layers have been widely used in a variety of deep models for up-sampling, including encoder-decoder networks for semantic segmentation and deep generative models for unsupervised learning. One of the key limitations of deconvolutional operations is that they result in the so-called checkerboard problem. This is caused by the fact that no direct relationship exists among adjacent pixels on the output feature map. To address this problem, we propose the pixel deconvolutional layer (PixelDCL) to establish direct relationships among adjacent pixels on the up-sampled feature map. Our method is based on a fresh interpretation of the regular deconvolution operation. The resulting PixelDCL can be used to replace any deconvolutional layer in a plug-and-play manner without compromising the fully trainable capabilities of original models. The proposed PixelDCL may result in slight decrease in efficiency, but this can be overcome by an implementation trick. Experimental results on semantic segmentation demonstrate that PixelDCL can consider spatial features such as edges and shapes and yields more accurate segmentation outputs than deconvolutional layers. When used in image generation tasks, our PixelDCL can largely overcome the checkerboard problem suffered by regular deconvolution operations.
Paper summary:
This paper proposes a technique to generalize deconvolution operations used in standard CNN architectures. Traditional deconvolution operation uses independent filter weights to compute output features at adjacent pixels. This work proposes to do sequential prediction of adjacent pixel features (via intermediate feature maps) resulting in more spatially smooth outputs for deconvolution layer. This new layer is referred to as ‘pixel deconvolution layer’ and it is demonstrated on two tasks of semantic segmentation and face generation.
Paper Strengths:
- Despite being simple technique, the proposed pixel deconvolution layer is novel and interesting.
- Experimental results on two different tasks demonstrating the general use of the proposed deconvolution layer.
Major Weaknesses:
- The main weakness of this paper lies in its weak experiments. Although authors say that several possibilities exist for the dependencies between intermediate feature maps, there are no systematic ablation studies on what type of connectivities work best for the proposed layer. Authors experimented with two randomly chosen connectivities which is not enough to understand what type of connectivities work best. This is important as this forms the main contribution of the paper.
- Also, several quantitative results seem incomplete. Why is the DeepLab-ResNet performance so low? A quick look at PascalVOC results indicate that DeepLab-ResNet has IoU of over 79 on this dataset, but the reported numbers in this paper are only around 73 IoU. There is no mention of IoU for base DeepLab-ResNet model and the standard DeepLab+CRF technique. And, there are no quantitative results on image generation.
Minor Weaknesses:
- Although the paper is easy to understand, several parts of the paper are poorly written. Several sentences are repeated multiple times across the paper. Some statements need corrections/refinements such as “mean IoU is a more accuracy evaluation measure”. And, it is better to under-tone some statements such as changing “solving” to “tackling”.
- The illustration of checkerboard artifacts from standard deconvolution technique is not clear. For example, the results presented in Figure-4 indicate segmentation mistakes of the network rather than checkerboard artifacts.
Clarifications:
- Why authors choose to ‘resize’ the images for training semantic segmentation networks, instead of generally used ‘cropping’ to create batches?
- I can not see the ‘red’ in Figure-5. I see the later feature map more as ‘pinkish’ color. It is probably due to my color vision. In any case, it is better to use different color scheme to distinguish.
Suggestions:
- I strongly advice authors to do some ablation studies on connectivities to make this a good paper. Also, it would be great if authors can revise the writing thoroughly to make this a more enjoyable read.
Review Summary:
The proposed technique, despite being simple, is novel and interesting. But, the weak and incomplete experiments make this not yet ready for publication.