Equivariant neural networks are special types of neural networks that preserve some symmetries on the data set. In this paper, we provide a method to modify a neural network to an equivariant one, which we call equivarification.
In this work, the authors employ concepts from group theory to turn an arbitrary feed forward neural network into an equivariant one, i.e. a network whose output transforms in a way that is consistent with the transformation of the input. To this end, the authors first introduce the basic concepts of group theory required to follow their work and provide a comprehensive definition of equivariance. They then explain how to equivarify (w.r.t. a finite group G) a given neural network, and present experimental results on rotated MNIST digits to support their approach.
I find that the proposed approach is very elegant and addresses a highly important question, namely how to devise or modify an architecture such that it preserves symmetries. In my opinion, the current paper makes an interesting theoretical contribution towards achieving this goal, but it also has several shortcomings that are detailed below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.
Major comments:
1) Scaling
The authors mention in the abstract that ‘although the network size scales up, the constructed equivariant neural network does not increase the complexity of the network compared with the original one in terms of the number of the parameters.’ Based on Eq. (3.0.2) and Fig. 3, it is my understanding that n evaluations of each data point (input to a layer) are required for a cyclic group of order n. If the outputs of a layer are then concatenated, the input dimension of the subsequent layer grows by a factor of n. I would therefore argue that out-the-box application of the proposed approach does increase the number of variables dramatically and that the abstract is misleading in this respect. The authors briefly comment on this point with one sentence in the third paragraph of the introduction. However, I would appreciate if this point was addressed in more detail, for example in a dedicated paragraph after the theory is introduced. Please also address the question of whether variable sharing is essential from an equivariance point of view, or whether it’s simply a necessity to prevent an explosion of the number of parameters. Furthermore, convolutions encode translational symmetry which may be beneficial for the current application but may not be desirable for other datasets. A few comments for clarification would be very helpful. Finally, the equivarified network seems to increase the required number of computations significantly compared to the original one, which I find worth a comment .
2) Experiment
The authors only consider a convolutional architecture and say that ‘in order to illustrate flexibility we choose not to simply equivarify the original network’. However, to me one of the main advantages of the paper seems to be that you can take this approach to equivarify any FFN. It would therefore be interesting to see this approach be applied to different networks starting with a simpler one, e.g. a 2-layer MLP. The authors could then compare the original network to the equivarified one with and without variable sharing. That would not only help the reader understand the approach better but also be much more in line with the main motivation of the paper. Then adding a second experiment, e.g. a convolutional architecture, to demonstrate flexibility would be very interesting. With regard to Fig. 4, I think there may be better ways of summarising the results than dumping 160 numbers of which only 4 seem to be of interest. The message seems to be that the network yields identical probabilities irrespective of the degree of rotation. What I find surprising is that all numbers are actually identical (shifted by 10). Is this by construction?
3) Limitations
As indicated in the second paragraph of Sec. 4, this approach is limited to finite groups and the authors only consider image rotations w.r.t. the cyclic group of degree 4. Although I appreciate that this is meant to serve as a toy problem to illustrate that the approach works, I do not think that rotations by a constant angle are very interesting. What would be really interesting is equivariance w.r.t continuous rotations (Lie Groups), e.g. the SO(2) in this particular case. I doubt that an extension to the SO(2) is straightforward within the current theoretical framework. However, even if that is the case, I would appreciate if the authors could comment on this in a paragraph.
Minor comments:
i) There are many typos and grammar mistakes in the paper:
‘any feedforward networks’ -> ‘any feedforward network’.
‘enables to design’ -> ‘enables us to design’
‘our proposed equivarification method use’ -> ‘our proposed equivarification method uses’
‘traditional sense multiplication’ -> ‘traditional sense of multiplication’
‘a group G acts’ -> ‘a group G that acts’
‘neural network that defined on the’ -> ‘neural network that is defined on the’
‘which achieves promising performance’ -> ‘which achieve promising performance’
‘supplymentary material’ -> ‘supplementary material’ 
Etc.
ii) I think there may be a mistake in the 3rd point of Definition 3.0.3: For consistency with the previous definitions and with Fig. 1, shouldn’t F map from X to Z and \hat F from X to \hat Z? 
iii) Last paragraph of Sec 3: ‘then after the identification \hat F becomes a map from Z to...'.  Should it be ‘a map from X to ..’?
iv) In Definition 3.0.3 you define the tuple (\hat Z, T, p) to be a G-equivarification, but in the paragraph below you call the G-product itself a G-equivarification (without including T and p). 
v) Footnote 2: You could correct for that and present the theory shifting by g instead of g^-1 to make it easier for the reader to follow. Or, at least, give a reference to the footnote earlier on in Example 4.0.1 to avoid confusion.
vi) Unless there is a special reason for this, I would suggest changing the enumeration of definitions, lemmas, examples and equations, i.e. (3.0.1) -> (3.1), etc...
*********************************************************
I increased my rating based on the authors addressing many of my comments. 
*********************************************************