POLAR TRANSFORMER NETWORKS
Convolutional neural networks (CNNs) are inherently equivariant to translation. Efforts to embed other forms of equivariance have concentrated solely on rotation. We expand the notion of equivariance in CNNs through the Polar Transformer Network (PTN). PTN combines ideas from the Spatial Transformer Network (STN) and canonical coordinate representations. The result is a network invariant to translation and equivariant to both rotation and scale. PTN is trained end-to-end and composed of three distinct stages: a polar origin predictor, the newly introduced polar transformer module and a classifier. PTN achieves stateof-the-art on rotated MNIST and the newly introduced SIM2MNIST dataset, an MNIST variation obtained by adding clutter and perturbing digits with translation, rotation and scaling. The ideas of PTN are extensible to 3D which we demonstrate through the Cylindrical Transformer Network.
This paper presents a new convolutional network architecture that is invariant to global translations and equivariant to rotations and scaling. The method is combination of a spatial transformer module that predicts a focal point, around which a log-polar transform is performed. The resulting log-polar image is analyzed by a conventional CNN.
I find the basic idea quite compelling. Although this is not mentioned in the article, the proposed approach is quite similar to human vision in that people choose where to focus their eyes, and have an approximately log-polar sampling grid in the retina. Furthermore, dealing well with variations in scale is a long-standing and difficult problem in computer vision, and using a log-spaced sampling grid seems like a sensible approach to deal with it.
One fundamental limitation of the proposed approach is that although it is invariant to global translations, it does not have the built-in equivariance to local translations that a ConvNet has. Although we do not have data on this, I would guess that for more complex datasets like imagenet / ms coco, where a lot of variation can be reasonably well modelled by diffeomorphisms, this will result in degraded performance.
The use of the heatmap centroid as the prediction for the focal point is potentially problematic as well. It would not work if the heatmap is multimodal, e.g. when there are multiple instances in the same image or when there is a lot of clutter.
There is a minor conceptual confusion on page 4, where it is written that "Group-convolution requires integrability over a group and identification of the appropriate measure dg. We ignore this detail as implementation requires application of the sum instead of integral."
When approximating an integral by a sum, one should generally use quadrature weights that depend on the measure, so the measure cannot be ignored. Fortunately, in the chosen parameterization, the Haar measure is equal to the standard Lebesque measure, and so when using equally-spaced sampling points in this parameterization, the quadrature weights should be one. (Please double-check this - I'm only expressing my mathematical intuition but have not actually proven this).
It does not make sense to say that "The above convolution requires computation of the orbit which is feasible with respect to the finite rotation group, but not for general rotation-dilations", and then proceed to do exactly that (in canonical coordinates). Since the rotation-dilation group is 2D, just like the 2D translation group used in ConvNets, this is entirely feasible. The use of canonical coordinates is certainly a sensible choice (for the reason given above), but it does not make an infeasible computation feasible.
The authors may want to consider citing
- Warped Convolutions: Efficient Invariance to Spatial Transformations, Henriques & Vedaldi.
This paper also uses a log-polar transform, but lacks the focal point prediction / STN.
Likewise, although the paper makes a good effort to rewiev the literature on equivariance / steerability, it missed several recent works in this area:
- Steerable CNNs, Cohen & Welling
- Dynamic Steerable Blocks in Deep Residual Networks, Jacobsen et al.
- Learning Steerable Filters for Rotation Equivariant CNNs, Weiler et al.
The last paper reports 0.71% error on MNIST-rot, which is slightly better than the PTN-CNN-B++ reported on in this paper.
The experimental results presented in this paper are quite good, but both MNIST and ModelNet40 seem like simple / toyish datasets. For reasons outlined above, I am not convinced that this approach in its current form would work very well on more complicated problems. If the authors can show that it does (either in its current form or after improving it, e.g. with multiple saccades, or other improvements) I would recommend this paper for publication.
Minor issues & typos
- Section 3.1, psi_gh = psi_g psi_h. I suppose you use psi for L and L', but this is not very clear.
- L_h f = f(h^{-1}), p. 4
- "coordiantes", p. 5