Spatial Transformer Networks
Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.
This work addresses the question of how to improve the invariance properties of Convolutional Neural Networks. It introduces the so-called spatial transformer, a layer that performs an adaptive warping of incoming feature maps, thus generalizing the recent attention mechanisms for images. The resulting model requires no extra supervision and is trained back-to-back using backpropagation, leading to state-of-the-art results on several classification tasks. The paper is clearly written and its main contribution, the spatial transformer layer, is valuable for its novelty, simplicity and effectiveness. The related work section covers most relevant literature, except perhaps recent works that combine deformable parts models with CNNs (see for example "Deformable Part Models are Convolutional Neural Networks", "End-to-End Integration of a Convolution Network, Deformable Parts Model and Non-Maximum Suppression" both at cvpr 2015), since they also incorporate an inference over deformation or registration parameters, as in the spatial transformer case. The numerical experiments cover some relatively challenging datasets (although not large scale tasks such as Imagenet or general object recognition) which highlight the gain produced by the registration layers. Here are some detailed comments:--> relationship with a general bilinear model. The spatial transformer applies a linear transform A(x,theta(x)) to the input feature map x, which is itself indexed by a low-dimensional vector of parameters theta(x) that are adapted to the input. This is a particular instance of a general bilinear map, in which a given family of transformations (warpings or affine maps) is considered. Looking at the experiments, it seems as if the transformer is asking for more capacity, since the richer class (thin plate spline) systematically improves over simpler models of transformations. One could then attempt to make the bilinear model a bit more expressive by replacing the class of warping operators by a learnt subfamily of linear operators. Another related comment is whether it would be interesting to consider 3D warpings across feature maps as well. That could perhaps allow the network to model relationships across features with extra capacity.  --> relationship with data augmentation. In order to increase the invariance/stability of the network to large displacements or transformations out of the scope of the pooling layers, one typically resorts to data augmentation. How does the spatial transformer relate to this method? can they work together? One could argue that data augmentation uses prior information, whereas the spatial transformer does not assume classes to be invariant to known transformations. --> Another comment is whether one could use the learnt registration parameters to construct adversarial examples/data augmentation. In the model, there is a relationship between the so-called localisation parameters theta(x) and x. If say x'= T_beta(x), where T_beta is assumed to belong to the family currently implemented by the spatial transformer,then it is reasonable to expect to have theta(x') = theta(x)+beta, assuming also a group structure in the space of transformations. In any case, the localisation net should satisfy T_{theta(x)} T_beta = T_{theta( T_beta(x) ) } for all x and beta. Could this property help regularize the localisation net? --> The paper presents arguments in favor of the spatial transformers together with experiments that show its efficiency. However, the authors do not mention any scenario where the method could fail, or even hurt the performance of vanilla CNNs. When does it fail, or what are its limitations? For instance, aliasing can be potentially dangerous, as pointed out briefly, but also bilinear interpolation is known to be a bad kernel if one needs to preserve high frequency content, which might be important in, say, texture classification or complex object recognition tasks.