An intriguing failing of convolutional neural networks and the CoordConv solution
Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x, y) Cartesian space and coordinates in one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious. We call this solution CoordConv, which works by giving convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv allows networks to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. CoordConv solves the coordinate transform problem with perfect generalization and 150 times faster with 10-100 times fewer parameters than convolution. This stark contrast raises the question: to what extent has this inability of convolution persisted insidiously inside other tasks, subtly hampering performance from within? A complete answer to this question will require further investigation, but we show preliminary evidence that swapping convolution for CoordConv can improve models on a diverse set of tasks. Using CoordConv in a GAN produced less mode collapse as the transform between high-level spatial latents and pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST detection showed 24% better IOU when using CoordConv, and in the Reinforcement Learning (RL) domain agents playing Atari games benefit significantly from the use of CoordConv layers.
Update following author response  Thank you for addressing many of the concerns. Overall, all my previous concerns about the quality, clarity and significance are alleviated. However I am not very certain about the originality still.  ------------------------  - summary of the main ideas  It was difficult to provide a concise summary of the main ideas because there is a lot going on in this paper. Note that various models were used for experimentation and that model specifications are given in the supplementary material. It was also difficult to keep track of which kind of convolutional neural networks (CNNs) were used for which subproblem.  First, this paper shows instances where CNNs fail at particular subsets of the coordinate transform problem. For "normal" CNNs, the paper shows:   1. that a fully convolutional model fails to solve the supervised coordinate classification problem and the supervised rendering problem.  2. that a convolutional model with max-pooling and fully connected layers fails to solve the supervised coordinate regression problem, specifically the quadrant split. However the respective model generalizes well for the supervised coordinate regression problem with the uniform split. 3. that a fully convolutional model achieves limited generalization on the supervised coordinate regression problem with quadrant split. The respective model fails on the supervised coordinate regression problem with uniform split.   Second, a solution called CoordConv is proposed to solve this problem, generalized as the coordinate transform problem. With CoordConv, the input to the CNN is augmented with 2 extra channels representing the x and y Cartesian coordinates respectively. This is the only modification needed to implement CoordConv. They show that using CoordConv, all the above mentioned toy problems are solved with extremely high accuracy, requiring far less training time and utilizing smaller CNNs (less model parameters).   Third, to show that the CoordConv solution can increase performance on other problems, and not just this toy problem, a comparison is made between "normal" CNNs and CoordConv CNNs on the following problems: - Unsupervised Density Learning with GANs - Image classification with ResNet-50 - Deep Reinforcement Learning (RL) on a selection of Atari games with the A2C network   - quality -- strengths .Performance on the toy problems was demonstrated very well and the results were impressive. .Figure 8: (a) I was surprised to see the great improvement in performance because MsPacman is notoriously difficult.  .In the RL experiment and the image classification task, significance testing was performed. -- weaknesses .In lines 22 and 72 it is mentioned that for the ImageNet classification with ResNet-50 the performance increase is significant, with p=0.051. This is simply a false claim, which you seem to realize as well, since you write in line 224 that this result is "nearly statistically significant".  .Figure 7: The results of the GAN and CoordConv GAN are ambiguous. For example, I see no difference between the two models in (a) and in (c) I can argue that the CoordConv is the one suffering from mode collapse, since the "normal" GAN shows a more diverse sampling. For (d) I don't know how to interpret this. .No quantifiable measure for mode collapse is used. It would have been much better if an objective measure for mode collapse had been used, such as entropy.  .Figure 8: (b) it is highly questionable that this graph is cut off at timestep 7, exactly when the overlap in performance happens. This result seems cherry-picked. Especially since (a) and (c) go up to timestep 8. Please explain why this was done, ideally provide a graph that goes to timestep 8. Also, why is there no table of all the Atari games together with their results? .No theoretical justification was given whatsoever for the behaviors exhibited by the various neural networks. The rigor of the experiments outside of the toy context was not satisfactory to a NIPS standard. .The role of max-pooling for achieving translation invariance is not addressed. For example, no explanation is given for the differences in performance between (2) and (3) in my summary above. To me it is clear that max-pooling will be problematic for tasks that require translation dependence, since max-pooling increases translation invariance. .In line 162 the paper claims to verify that "performance on this task does in fact predict performance on larger problems". However, no such verification is presented. .The reasoning for the demonstration on the RL task and the image classification task is not given. I am left with the question of "why would these specific tasks benefit from CoordConv?". To me it seems that a location specific task would benefit from CoordConv, however with the image classification I don't see the usefulness of CoordConv because ImageNet classification task is translation invariant.   - clarity -- strengths .Mostly very good visuals, clearly representing the benefit of CoordConv on the toy problems. .Paper was well structured -- weaknesses .It was difficult to keep track of which kind of convolutional neural networks (CNNs) were used for which subproblem. A table showing an overview of the various models would have been very useful. .The paper does not provide enough information for replication. Even though it is mentioned how the grid search for parameters is performed, it is not clear which final networks were used to obtain the figures. .Figure 7 did not give a clear overview of the differences between the models.  .It is not clear to me how to implement the CoordConv solution in the supervised coordinate classification problem and the supervised rendering problem. In Figure S1, what does "32-32-64-64-1" mean?  - originality -- strengths .This is the first time that I have seen such an approach to expand the capabilities of a CNN towards translation dependence.  -- weaknesses .I am not entirely sure how original this contribution is, given that there is another paper which uses a similar approach of channel coordinate augmentation in their pipeline to perform image inpainting, see Figure 2 in the supplementary material: Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. "Deep image prior." arXiv preprint arXiv:1711.10925 (2017). Supplementary material: https://box.skoltech.ru/index.php/s/ib52BOoV58ztuPM#pdfviewer However said paper does not make this the main focus of their work and it is also not clear to me if the "meshgrid" channels are directly added to the input.  - significance -- strengths .The failing of traditional CNNs (with max pooling or fully convolutional) has been adequately demonstrated on the toy problems. It is also clear that the CoordConv solution solves these problems. -- weaknesses .However, it is not clear to me that this solution is useful beyond a toy context.