It has been repeatedly observed that convolutional architectures when applied to image understanding tasks learn oriented bandpass filters. A standard explanation of this result is that these filters reflect the structure of the images that they have been exposed to during training: Natural images typically are locally composed of oriented contours at various scales and oriented bandpass filters are matched to such structure. The present paper offers an alternative explanation based not on the structure of images, but rather on the structure of convolutional architectures. In particular, complex exponentials are the eigenfunctions of convolution. These eigenfunctions are defined globally; however, convolutional architectures operate locally. To enforce locality, one can apply a windowing function to the eigenfunctions, which leads to oriented bandpass filters as the natural operators to be learned with convolutional architectures. From a representational point of view, these filters allow for a local systematic way to characterize and operate on an image or other signal.
This paper claims that convolutional filters in CNNs are not the result of fitting to the input data distribution but they are the optimal solution to a spectral decomposition of the convolutional operator. 
Positive things about this work:
1) it is clearly written
2) the fact that Gabor wavelets are the eigenfunctions of convolution is sound.
3) it provides good food for thought about what needs to be learned and what comes from the pre- specified choice of architecture
Negative things about this work:
1) it misses references to relevant works. In particular, there is an article making essentially the same points:
Joan Bruna, Soumith Chintala, Yann LeCun, Serkan Piantino, Arthur Szlam, and Mark Tygert, "A mathematical motivation for complex-valued convolutional networks," Neural Computation, 28 (5): 815-825, 2016
http://tygert.com/ccnet.pdf
where these authors make the same conclusions and observe that learning reduces to figuring out the  windowing, number of scales, etc. but not the type of filters.
This prior work greatly reduces the impact of this contribution, unfortunately.
There are other references that are missing, but these are minor points compared to the above. For instance, I'd recommend to cite Hubel & Wiesel's seminal work on mapping the mammalian receptive fields, and older work by M. Lewicki about analyzing learned receptive fields by sparse coding algorithms, similarly to the cited B. Olshausen et al.  
2) The Authors show that Gabor wavelets are eigenfunction of convolutions and they stop here to conclude that filters in CNNs are the way they are because of the architecture, but how about the effect of the non-linearities, depth and the type of cost used for training? The work is unfinished without a thorough analysis and discussion of these crucial aspects.