We flip the usual approach to study invariance and robustness of neural networks by considering the non-uniqueness and instability of the inverse mapping. We provide theoretical and numerical results on the inverse of ReLU-layers. First, we derive a necessary and sufficient condition on the existence of invariance that provides a geometric interpretation. Next, we move to robustness via analyzing local effects on the inverse. To conclude, we show how this reverse point of view not only provides insights into key effects, but also enables to view adversarial examples from different perspectives.
Review
This paper discusses invariances in ReLU networks. The discussion is anchored around the observation that while the spectral norm of neural networks layers (their product bounds the Lipschitz constant) has been investigated as a measure of robustness of nets, the smallest singular values are also of interest as these indicate directions of invariance. 
The paper consists mostly of a theoretical analysis with little empirical support, focusing on a property of matrices called omnidirectIonality. The definition given seems weird — an A \in R^{m \times n} is omnidirectional if there exists a unique x \in R^n such that Ax \leq 0. 
If there is a *unique* x then that x must be 0. Else if there were a nonzero x for which Ax \leq 0, then A(cx) also \leq 0 for any positive scalar 0 and thus x is not unique. Moreover if x must be equal to 0 Ax \leq 0 and at that point Ax = 0, then that means there exists no x for which Ax < 0, so why not just say this outright? Perhaps a cleaner definition would just be “A is full rank and there does not exist any X such that Ax < 0? Also perhaps better to use the curly sign for vector inequality. 
Overall the paper, while interesting is unacceptably messy. 
The first two pages have no paragraph breaks!!! This means either that the author are separating paragraphs with \\ \noindent or that they have modified the style file to remove paragraph breaks to save space. Either choice is unreadable and unacceptable. The paper is also littered with typos and vague statements (many enumerated below under *small issues*). In this case, they add up to make a big issue. 
The notation at the top of page 4 — see (1) and (2) — comes out of nowhere and requires explanation. |_{y>0} x + b |_{y>0}  <— what is the purpose of the subscripts here? Why is this notation never introduced?
Ultimately this paper focuses on the question on whether the pre-image of a ReLU layer can be concluded (based on the post-image) to be a singleton,  a compact polytope, or if it has infinite volume. The paper offers some analysis, suggesting when each of the conditions occurs, upper bounds the smallest singular value of D A (where the example dependent diagonal matrix D incorporates the ReLU activation (shouldn’t this be more clearly introduced and notated?). 
Ultimately this paper is interesting but falls well below the standards of exposition that I expect from a theory paper and doesn’t go very far at connecting the analysis back to the claimed motivation of investigating practical invariances. If the authors significantly improve the quality of the draft, I’ll be happy to revisit it and re-evaluate my score. 
Small issues
The following is a *very* incomplete list of small bugs found in the paper:
“From a high-level perspective both of these approaches” --> missing comma after “perspective”
"as well as the gradient correspond to the highest
possible responds for a given perturbation" --> incomprehensible "corresponding?" "possible responds?" do you mean "response", and if so what is the precise technical meaning here?
"analyzing the lowest possible response" what does "response' mean here?
"We provide upper bounds on the smallest singular value" -- the singular value of what? This hasn't been stated yet.
"reverse view on adversarial examples" --- what this means isn't clear from the preceding text.
"we aim to theoretically derive means to uncover mechanisms of rectifier networks without assumptions on the weights" -- what does "mechanisms" mean here?
Notation section -- need a sentence here at the beginning, can't just have a section heading followed by bullets. 
"realated"