SPHERICAL CNNS ON UNSTRUCTURED GRIDS
We present an efficient convolution kernel for Convolutional Neural Networks (CNNs) on unstructured grids using parameterized differential operators while focusing on spherical signals such as panorama images or planetary signals. To this end, we replace conventional convolution kernels with linear combinations of differential operators that are weighted by learnable parameters. Differential operators can be efficiently estimated on unstructured grids using one-ring neighbors, and learnable parameters can be optimized through standard back-propagation. As a result, we obtain extremely efficient neural networks that match or outperform state-of-the-art network architectures in terms of performance but with a significantly smaller number of network parameters. We evaluate our algorithm in an extensive series of experiments on a variety of computer vision and climate science tasks, including shape classification, climate pattern segmentation, and omnidirectional image semantic segmentation. Overall, we (1) present a novel CNN approach on unstructured grids using parameterized differential operators for spherical signals, and (2) show that our unique kernel parameterization allows our model to achieve the same or higher accuracy with significantly fewer network parameters.
The paper presents a new convolution-like operation for parameterized manifolds, and demonstrates its effectiveness on learning problems involving spherical signals. The basic idea is to define the MeshConvolution as a linear combination (with learnable coefficients) of differential operators (identity, gradient, and Laplacian). These operators can be efficiently approximated using the 1-hop neighbourhood of a vertex in the mesh.
In general I think this is a strong paper, because it presents a simple and intuitive idea, and shows that it works well on a range of different problems. The paper is well written and mostly easy to follow. The appendix contains a wealth of detail on network architectures and training procedures.
What is not clear to me is how exactly the differential operators are computed, and how the MeshConvolution layer is implemented. The authors write that "differential operators can be efficiently computed using Finite Element basis, or derived by Discrete Exterior Calculus", but no references or further detail is provided. The explanation of the derivative computation is:
"The first derivative can be obtained by first computing the per-face gradients, and then using area-weighted average to obtain per-vertex gradients. The dot product between the per-vertex gradient value and the corresponding x and y vector fields are then computed to acquire grad_x F and grad_y F."
What are per-face gradients and how are they computed? Is the signal sampled on vertices or on faces? What area is used for weighting? What is the exact formula? What vector fields are you referring to? (I presume these are the coordinate vector fields). In eq. 5, what are F_i and F_j? What is the intuition behind the cotangent formula (eq. 5), and where can I read more? etc.
Please provide a lot more detail here, delegating parts to an appendix if necessary. Providing code would be very helpful as well.
A second (minor) concern I have is to do with the coordinate-dependence of the method. Because the MeshConvolution is defined in terms of (lat / lon) coordinates in a non-invariant manner, and the sphere does not admit a global chart, the method will have a singularity at the poles. This is confirmed by the fact that in the MNIST experiment, digits are rotated to the equator "to prevent coordinate singularity at the poles". I think that for many applications, this is not a serious problem, but it would still be nice to be transparent and mention this as a limitation of the method when comparing to related work.
In "Steerable CNNs", Cohen & Welling also used a linear combination of basis kernels, so this could be mentioned in the related work under "Reparameterized Convolutional Kernel".
To get a feel for the differential operators, it may be helpful to show the impulse response (at different positions on the sphere if it matters).
In experiment 4.1 as well as in the introduction, it is claimed that invariant/equivariant models cannot distinguish rotated versions of the same input, such as a 6 and a 9. Although indeed an invariant model cannot, equivariant layers do preserve the ability to discriminate transformed versions of the same input, by e.g. representing a 9 as an upside-down 6. So by replacing the final invariant pooling layer and instead using a fully connected one, it should be possible to deal with this issue in such a network. This should be mentioned in the text, and could be evaluated experimentally.
In my review I have listed several areas for improvement, but as mentioned, overall I think this is a solid paper.