DEEP, SKINNY NEURAL NETWORKS ARE NOT UNI- VERSAL APPROXIMATORS
In order to choose a neural network architecture that will be effective for a particular modeling problem, one must understand the limitations imposed by each of the potential options. These limitations are typically described in terms of information theoretic bounds, or by comparing the relative complexity needed to approximate example functions between different architectures. In this paper, we examine the topological constraints that the architecture of a neural network imposes on the level sets of all the functions that it is able to approximate. This approach is novel for both the nature of the limitations and the fact that they are independent of network depth for a broad family of activation functions.
This is a very nice paper contributing to what I consider a relatively underexplored but potentially very promising research direction. The title of the paper in my opinion undersells the result which is not only that "deep skinny neural networks" are not universal approximators, but that the class of functions which cannot be approximated includes a set of practically relevant classifiers as illustrated by the figure on page 8. The presentation is extremely clear with helpful illustrations and toy but insightful experiments.
My current rating of this paper is based on assuming that the following concerns will be addressed. I will adjust the score accordingly after authors' reply.
Main:
- A very similar result can be found in Theorem 7 of Beise et al.'s "On decision regions of narrow deep neural networks" from July 2018 ( https://arxiv.org/abs/1807.01194 )
	Some differences:
		- The other paper considers connected whereas this paper considers path-connected components (the former is more general).
		- The other paper only considers multi-label classification, this paper is relevant to all classification and regression problems (the latter is more general).
		- The other paper requires that the activation function is "strictly monotonic or ReLU" whereas this paper allows "uniformly approximable with one-to-one functions" activations (the latter is more general).
	The result in this paper seems slightly more general but largely similar. Can you please comment on the differences/relation to the other paper?
- Proof of Lemma 4:  "Thus the composition \hat{f} is also one-to-one, and therefore a homeomorphism from R^n onto its image I_{\hat{f}}". Is it not necessary that \hat{f} has a continuous inverse in order to be a homeomorphism? I do not immediately see whether the class of activation functions considered in this paper implies that this condition is satisfied. Please clarify. 
Minor:
- Proof of Lemma 5: It seems g is assumed to be continuous at several places (e.g. "... level sets of are closed as subsets of R^n ..." seems to assume that pre-image of a closed set under g is closed, or later "This implies g(F) is a compact subset of R ..."). Perhaps you are assuming that M is a set of continuous functions and using the fact that uniform limit of continuous functions is continuous? Please clarify.
- On p.4: "This is fairly immediate from the assumptions on \varphi and the fact that singular transition matrices can be approximated by non-singular ones." Is the second part of the sentence using the assumption that the input space is compact? Please clarify.
- Second line in Section 5: i < k should probably be i < \kappa.