DEEP LEARNING 3D SHAPES USING ALT-AZ ANISOTROPIC 2-SPHERE CONVOLUTION
The ground-breaking performance obtained by deep convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to extend it for 3D geometric tasks. One of the main challenge in applying CNNs to 3D shape analysis is how to define a natural convolution operator on noneuclidean surfaces. In this paper, we present a method for applying deep learning to 3D surfaces using their spherical descriptors and alt-az anisotropic convolution on 2-sphere. A cascade set of geodesic disk filters rotate on the 2-sphere and collect spherical patterns and so to extract geometric features for various 3D shape analysis tasks. We demonstrate theoretically and experimentally that our proposed method has the possibility to bridge the gap between 2D images and 3D shapes with the desired rotation equivariance/invariance, and its effectiveness is evaluated in applications of non-rigid/ rigid shape classification and shape retrieval.
# Summary
This paper proposes a new kind of spherical convolution for use in spherical CNNs, and evaluates it on rigid and non-rigid 3D shape recognition and retrieval problems. Previous work has either used general anisotropic convolution or azimuthally isotropic convolution. The former produces feature maps on SO(3), which is deemed undesirable because processing 3-dimensional feature maps is costly. The latter produces feature maps on the sphere, but requires that filters be circularly symmetric / azimuthally isotropic, which limits modeling capacity. This paper proposes an anisotropic spherical convolution that produces 2D spherical feature maps. The paper also introduces an efficient way of processing geodesic / icosahedral spherical grids, avoiding complicated spectral algorithms.
# Strengths
The paper has several strong points. It is well written, clearly structured, and the mathematics is clear and precise while avoiding unnecessary complexity. Much of the relevant related work is discussed, and this is done in a balanced way. Although it is not directly measured, it does seem highly likely that the alt-az convolution is more computationally efficient than SO(3) convolution, and more expressive than isotropic S2 convolution. The most important contribution in my opinion is the efficient data structure presented in section 4, which allows the spherical convolution to be computed efficiently on GPUs for a grid that is much more homogeneous than the lat/lon grids used in previous works (which have very high resolution near the poles, and low resolution at the equator). The idea of carving up the icosahedral grid in just the right way, so that the spherical convolution can be computed as a planar convolution with funny boundary conditions, is very clever, elegant, and practical.
# Weaknesses
There is however a misunderstanding about the properties of the alt-az convolution that must be cleared up before this paper can be published. To start with, the set of rotations R(phi, nu, 0) called the alt-az group in this paper is not a group in the mathematical sense. This easy to see, because a composition of rotations of the form Rz(phi) Ry(nu) is not generally of that form. For instance we can multiply Rz(phi) Ry(nu) by the element Rz(omega)Ry(0) = Rz(omega), which gives the element Rz(phi) Ry(nu) Rz(omega). As noted in the paper, this is a general element of SO(3) (and hence not in the set of alt-az rotations). So the closure axiom of a group is violated.
This matters, because the notion of equivariance really only makes sense for a group. If a layer l satisfies l R = R l  (for R a alt-az rotation), then it automatically satisfies l RR' = RR' l, which means l is equivariant to the whole group generated by the set of alt-az rotations. As we saw before, this is the whole rotation group. This would mean that the layer is actually SO(3)-equivariant, but it has been proven [1], that any rotation equivariant layer between scalar spherical feature maps can be expressed as an azimuthally isotropic convolution. Since the alt-az convolution is not isotropic and maps between scalars on S2, it cannot be equivariant. This also becomes apparent in the experiments section, where rotational data augmentation is found to be necessary. The paper does not contain an attempted proof of equivariance, and if one tries to give one, the impossibility of doing so will become apparent.
I note that the alt-az convolution *is* equivariant to rotations in the subgroup SO(2) of rotations around the Z-axis.
Another somewhat jarring fact about the alt-az convolution is that it is not well defined on the south pole. The south pole can be represented by any pair of coordinates of the form phi in [0, 2pi], nu = +/- pi. But it is easy to see that eq. 10 will give different results for each of these coordinates, because they correspond to different rotations of the filter about the Z-axis. This is ultimately due to the fact that the set of alt-az rotations is not the same as the set of points on the sphere, topologically speaking. The set of points on the sphere can only be viewed as the quotient SO(3)/S(2).
The paragraph motivating the alt-az convolution on page 4 is not very clear, and some claims are questionable. I agree that local SO(2) invariance is too limiting. But it is not true that rotating filters is not effective in planar/volumetric CNNs, as shown by many recent papers on equivariant networks. I would suggest rewriting this paragraph to make it clearer and less speculative, and acknowledge that although rotating filters might increase computational complexity, it has often been shown very effective.
# Other comments
The experiments show that the method is quite effective. For instance, the SHREC17 results are on par with Cohen et al. and Esteves et al., presumably at a significantly reduced computational cost. That they do not substantially outperform these and other methods is likely due to the input representation, which is lossy, leading to a maximal performance shared by all three methods. An application to omnidirectional vision might more clearly show the strength of the method, but this would be a lot of work so I do not expect the authors to do that for this paper.
It would be nice to see a more direct comparison between the three definitions of spherical convolution (general SO3, isotropic S2, and anisotropic S2). Right now, the numbers reported in Cohen et al. and Esteves et al. are copied over, but there are probably many differences between the precise setup and architectures used in these papers. It would be interesting to see what happens if one uses the same architecture on a number of problems, changing only the convolution in each case.
Initially, I was a bit puzzled about why SO(3) augmentation seems to reduce accuracy in table 1. I think this is because SO(3) augmentation actually makes the classification problem harder if the input is initially aligned. Some more explanation / discussion would be good. 
It would be nice to explain the spherical parameterization in more detail. Is this operation itself rotation equivariant? 
Typos & minor issues
- Abstract: "to extract non-trivial features". The word non-trivial really doesn't add anything here. Similarly "offers multi-level feature extraction capabilities" is almost meaningless since all DL methods can be said to do so.
- Below eq. 5, D_R^{-1} should equal D_R(-omega, -nu, -phi). The order is reversed when inverting.
- "Different notations of convolutions" -> notions
- "For spherical functions there is no consistent and well defined convolution operators." As discussed above, the issue is quite a bit more subtle. There are exactly two well-defined convolution operators, but they have some characteristics deemed undesirable by the authors.
- "rationally symmetric" -> rotationally
- "exact hierarchical spherical patterns" -> extract
- It seems quite likely that the unpacking of the icosahedral/hexagonal grid as done in this paper has been studied before in other fields. References would be in order. Similarly, hexagonal convolution has a history in DL and outside.
- Bottom of page 7, capitalize "for".
- "principle curvatures" -> principal.
- "deferent augmentation modes" -> different
- "inspite" -> in spite
- "reprort" -> report
- "utlize" -> utilize
- "computer the convolution" -> compute
# Conclusion
Although the alt-az convolution lacks the mathematical elegance of the general anisotropic and azimuthally isotropic spherical convolutions, it still seems like a practically useful operation for some kinds of data, particularly when implemented using the homogeneous icosahedral/hexagonal grid and fast algorithm presented in this paper. Hence, I would wholeheartedly recommend acceptance of this paper if the authors correct the factual errors (e.g. the claim of SO(3)-equivariance) and provide a clear discussion of the issues. For now I will give an intermediate rating to the paper.
[1] Kondor, Trivedi, "On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups"