Encoding the scale information explicitly into the representation learned by a convolutional neural network (CNN) is beneficial for many vision tasks especially when dealing with multiscale inputs. We study, in this paper, a scaling-translationequivariant (ST -equivariant) CNN with joint convolutions across the space and the scaling group, which is shown to be both sufficient and necessary to achieve STequivariant representations. To reduce the model complexity and computational burden, we decompose the convolutional filters under two pre-fixed separable bases and truncate the expansion to low-frequency components. A further benefit of the truncated filter expansion is the improved deformation robustness of the equivariant representation. Numerical experiments demonstrate that the proposed scaling-translation-equivariant networks with decomposed convolutional filters (ScDCFNet) achieves significantly improved performance in multiscale image classification and better interpretability than regular CNNs at a reduced model size.
*Paper summary*
The authors propose a CNN architecture, that is theoretically equivariant to isotropic scalings and translations. For this they add an extra scale-dimension to activation tensors, along with the existing two spatial dimensions. In practice they implement this with scale-steerable filters, which are discretised and truncated in both the spatial and scale-dimensions. They also provide a deformation robustness analysis.
*Paper decision*
Thank you for writing a very interesting paper indeed. I have to admit I am somewhat on the fence about this paper. I think it contains many nice ideas, but the experimental section is somewhat lacking in terms of comparisons or insights which I can gain, and the theory has some missing elements too (which I shall discuss below). For that reason, I am recommending a weak reject, but would very easily upgrade this if the authors provide strong rebuttal to my comments below.
*Supporting arguments*
-Experiments: The experiments are quite light, although I must admit that many other works in the area of equivariance are also light on experimentation and if there is enough theory, that is not such a great issue. The main issues I have are 1) the choice of experiments, 2) the comparisons against an insufficient number of baselines, and 3) the ablation studies.
1) The choice of experiments: I think looking at scaled-MNIST is not particularly useful as an experiment nowadays, unless used as a toy experiment. A larger dataset with real-life scale variations would have been better. Furthermore, I’m not sure what the image reconstruction task is supposed to tell the reader, that the ScDCFNet is able to generalise to new scales?
2) There is a lot of concurrent work on multiscale architecture. To name a few:
Multigrid Neural Architectures, Ke et al., 2017
Feature Pyramid Networks for Object Detection, Lin et al., 2017
Multi-Scale Dense Networks for Resource Efficient Image Classification, Huang et al., 2018
Deep Scale-spaces, Worrall and Welling, 2019
I believe these works should at least be cited, but ideally compared against.
3) I would have liked to have seen some numerical results for the verification of scale equivariance. In the first layer the equivariance improbably going to be close to perfect because there is no truncation of the scale-dimension in the network, but after this layer I predict the equivariance error increases due to truncations effects. This would be a similar effect to other works, such as “Deep Scale-spaces” (Worrall and Welling, 2019).
-Theory: I think the theory is very interesting and a meaningful contribution in its own right. The authors treat scale-translation in continuous space as a group action on signals. This motivates the convolution presented in Theorem 1. This is a group convolution as per Cohen and Welling, (2016) modified to continuous space for a non-compact group. (Actually, this should be mentioned in the text as a matter of good scholarship). This is nice, since the group convolution has not been used in for a regular representation of a non-compact group (other than translation) as far as I can tell. What is perhaps not clear for me is how the theory breaks down in practice, since the implementation requires discretisation in space AND scale, which is not discussed much and furthermore, filters are restricted in spatial AND scale dimensions, leading to truncation errors in the equivariance. This last perspective was not discussed, and I feel it rather should be.
The theory goes further into deformation stability, which is a fresh perspective in the equivariance literature, so I am happy for its inclusion. Perhaps more motivation for why you think this is necessary would be warmly welcomed. 
*Smaller questions/notes for the authors*
- Technically this is scale-translation equivariance, you even write this in the method section of your paper, why is it not in the title? The reason I mention this Is because there are scale equivariant networks, which are not translation equivariant in the literature, see “Warped Convolutions: Efficient Invariance to Spatial Transformations” (Henriques and Vedaldi, 2019).
- Please make the link between Theorem 1 and the group convolution of Cohen and Welling (2016)
- Last paragraph of page 1: A steerable-in-scale filter does exist, see “Deformable kernels for early vision” (Perona, 1991)
- Please use numbering for all display-mode equations.
- This scheme works perfectly in the continuous-image setting, but how about for discretized images? In that case it cannot be scale-translation equivariant because the scaling-action is no longer part of a group.
- In equations 6 and 7, what is the specific motivation for using the laplacian eigen-decompositions as a basis? Is it for steerability with respect to the scale-translation action? Otherwise, surely any basis will do?
- Remark 2: If you are considering a truncation of the scale-axis, surely you can still use an L2 norm when quantifying the robustness of your representation?
- Bandlimiting of the filters: I would consider citing “Structured Receptive Fields in CNNs”, (Jacobsen et al., 2016)
- Pooling: A useful citation here would be “Making Convolutional Networks Shift-Invariant Again” (Zhang, 2019). They precise low pass filter before pooling.
- Experiments: 
—I’m not clear on the reason to include reasons with and without batch normalization. This is quite unconventional
— Did you ever use scale augmentation? What is the effect of training on one scale and then testing on another?