In this paper we propose autoencoder architectures for learning a Cohen-Welling (CW)-basis for images and their rotations. We use the learned CW-basis to build a rotation equivariant classifier to classify images. The autoencoder and classifier architectures use only tensor product nonlinearity. The model proposed by Cohen & Welling (2014) uses ideas from group representation theory, and extracts a basis exposing irreducible representations for images and their rotations. We give several architectures to learn CW-bases including a novel coupling AE architecture to learn a coupled CW-bases for images in different scales simultaneously. Our use of tensor product nonlinearity is inspired from recent work of Kondor (2018a). Our classifier has very good accuracy and we use fewer parameters. Even when the sample complexity to learn a good CW-basis is low we learn classifiers which perform impressively. We show that a coupled CW-bases in one scale can be deployed to classify images in a classifier trained and tested on images in a different scale with only a marginal dip in performance.
Recently there has been a spate of work on generalized CNNs that are equivariant to various symmetry groups, such a 2D and 3D rotations, the corresponding Euclidean groups (comprising not just rotations but also translations) and so on. The approach taken in most of the recent papers is to explicitly build in these equivariances by using the appropriate generalization of convolution. In the case of nontrivial groups this effectively means working in Fourier space, i.e., transforming to a basis that is adapted to the group action. This requires some considerations from represntation theory. 
Earlier, however, there was some less recognized work by Cohen and Welling on actually learning the correct basis itself from data. The present paper takes this second approach, and shows for a simple task like rotated MNIST, the basis can be learned from a remarkably small amount of data, and actually performs even better than some of the fixed basis methods. There is one major caveat: the nonlinearity itself has to be rotation-covariant, and for this purpose they use the recently introduced tensor product nonlinearities. 
The paper is a little rough around the edges. In the first 4 pages it launches into an exposition of ideas from representation theory which is too general for the purpose: SO(2) is a really simple commutative group, so the way that "tensor product" representations reduce to irreducibles could be summed up in the formula  $e^{-2\pi i k_1 x}e^{-2\pi i k_2 x}=e^{-2\pi i (k_1+k_2) x}$. I am not sure why the authors choose to use real representations (maybe because complex numbers are not supported in PyTorch, but this could easily be hacked) and I find that the real representations make things unnecessarily complicated. I suspect that at the end of the day the algorithm does something very simple (please clarify if working with 
real representations is somehow crucial). 
But this is exactly the beauty of the approach. The whole algorithm is very rough, there are only two layers (!), no effort to carefully implement nice exact group convolutions, and still the network is as good as the competition. Another significant point is that this network is only equivariant to rotations and not translations. 
Naturally, the question arises why one would want to learn the group adapted basis, when one could just compute it explicitly. There are two interesting lessons here that the authors could emphasize more:
1. Having a covariant nonlinearity is strong enough of a condition to force the network to learn a group adapted (Cohen-Welling) basis. This is interesting because Fourier space ("tensor") nonlinearities are a relatively new idea in the literature. This finding suggests that the nonlinearity might actually be more important than the basis.
2. The images that the authors work on are not functions on R^2, but just on a 28x28 grid. Rotating a rasterized image with eg. scikit-rotate introduces various artifacts. Similarly, going back and forth between a rasterized and polar coordinate based representation (which is effectively what would be required for "Harmonic Networks" and other Fourier methods) introduces messy interpolation issues. Not to mention downsampling, which is actually addressed in the paper. If a network can figure out how to best handle these issues from data, that makes things easier.
The experiments are admittedly very small scale, although some of the other publications in this field also only have small experiments. At the very least it would be nice to have standard deviations on the results and some measure of statistical significance. It would be even nicer to have some visualization of the learned bases/filters, and a bare bones matrix-level very simple description of the algorith. Again, what is impressive here is that such a small network can learn to do this task reasonably well.
Suggestions: 
1. Also cite the Tensor Field Networks of Thomas et al in the context of tensor product nonlinearities.
2. Clean up the formatting. "This leads us to the following" in a line by itself looks strange. Similarly "Classification ising the learned CW-basis". I think something went wrong with \itemize in Section 3.1.