We address the following question: How redundant is the parameterisation of ReLU networks? Specifically, we consider transformations of the weight space which leave the function implemented by the network intact. Two such transformations are known for feedforward architectures: permutation of neurons within a layer, and positive scaling of all incoming weights of a neuron coupled with inverse scaling of its outgoing weights. In this work, we show for architectures with non-increasing widths that permutation and scaling are in fact the only function-preserving weight transformations. For any eligible architecture we give an explicit construction of a neural network such that any other network that implements the same function can be obtained from the original one by the application of permutations and rescaling. The proof relies on a geometric understanding of boundaries between linear regions of ReLU networks, and we hope the developed mathematical tools are of independent interest.
In this paper, the authors studied the equivalence class of ReLU networks with non-increasing weights, and proved that permutation and scaling are the only function preserving weight transformations. The proof technique is novel, and provides some insights in the geometry space of the loss surface. I think the proof technique could have its general implications to some other research, however, the direct value of this paper is not very clear.
1)	The paper starts with the discussions on the redundancy introduced by over-parametrization, as one of its motivations. However, what is discussed in this paper is actually far distant from over-parametrization. The redundancy in over-parametrized networks and the redundancy in ReLU networks are different concepts and the connection between them is not well established. The over-parametrization is talking about the smooth information flow brought by wide intermediate layers, however, the equivalence class in this paper is more about the mathematical properties of the activation functions and the topological connections of the neural networks. I feel that the authors have some confusing understanding on these two concepts.
2)	The theory was established regarding restrictive types of ReLU networks (feedforward, with non-increasing width). However, many widely used networks are not of this kind. Without extensions to other shapes of the networks, convolutional and recurrent structures, and many kind of normalization transformations (e.g., batch norm and layer norm), the practical value of this paper is limited. 
3)	Some important references are missing. The following paper has in-depth theoretical analysis on ReLU networks, and characterizes the redundancy brought by positive-scale invariant properties of ReLU networks. It is strange that the authors did not cite it.  It would be necessary for the authors to discuss their additional technical contributions given this ICLR 2019 paper.
-  Q. Meng, et al. G-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space, ICLR 2019
4)	The practical implication of the theories in this paper is not clearly discussed. What if we know there are only these two kinds of redundancies? What kind of new algorithms and practices can be inspired by such theoretical understandings.
** I read the author rebuttal. Different people may have different criteria on evaluating a paper and my criterion is not only about "what", but more importantly about "so what". I still think the authors should think harder about the implications of their work, either theoretically or practically. Furthermore, I understand that some published papers also use restricted settings to ease their proofs, however, I personally do not think this is well justified. I think for top conference like ICLR, more solid and less restrictive works are preferred.  I could at most adjust my score to weak reject.