The classical Universal Approximation Theorem certifies that the universal approximation property holds for the class of neural networks of arbitrary width. Here we consider the natural 'dual' theorem for width-bounded networks of arbitrary depth. Precisely, let n be the number of inputs neurons, m be the number of output neurons, and let ρ be any nonaffine continuous function, with a continuous nonzero derivative at some point. Then we show that the class of neural networks of arbitrary depth, width n + m + 2, and activation function ρ, exhibits the universal approximation property with respect to the uniform norm on compact subsets of R n . This covers every activation function possible to use in practice; in particular this includes polynomial activation functions, making this genuinely different to the classical case. We go on to consider extensions of this result. First we show an analogous result for a certain class of nowhere differentiable activation functions. Second we establish an analogous result for noncompact domains, by showing that deep narrow networks with the ReLU activation function exhibit the universal approximation property with respect to the p-norm on R n . Finally we show that width of only n + m + 1 suffices for 'most' activation functions.
This paper proves universal approximation theorems for fully connected networks with fixed width and unbounded depth. Unlike recent results focusing on ReLU networks approximating scalar valued target functions, this paper provides universal approximation theorems for a wide range of activation functions and vector valued target functions. 
This paper provides a number of universal approximation theorems on different activation functions, but the central result can be stated as follows: 
Given input dimension n and output dimension m and activation \rho, assume \rho is a continuous function and there exists \alpha \in R such that \rho is continuously differentiable at \alpha and the derivative \rho’ is nonzero at \alpha. Then, a fully connected network with layer width n+m+2 and unbounded depth can approximate any continuous function on a compact domain to arbitrary sup-norm accuracy.
I would like to vote for acceptance of this paper because the authors develop nontrivial techniques that extend existing universal approximation results on width-bounded ReLU networks to essentially “all” other activation functions. I think this paper is well-written and well-organized; it gives a good overview of the existing results that contextualizes this paper well, briefly summarizes the main results, and then reveals the details of construction. I haven’t checked the full details of the proofs in the appendix, but as far as I can tell they look correct.
While I think that Section 4 reveals the proof techniques reasonably well, I believe that proofs of Propositions 4.3 and 4.7 should be covered/sketched in greater detail in the main text. For example, Theorem 4.4 builds upon Proposition 4.3 by approximating identity function locally using \rho. The main text only reveals approximation of identity function by \rho and defers the whole proof of Proposition 4.3 to the appendix. I think adding proof sketches for the propositions will be more helpful to the readers, and to this end, cutting down some text in the recurring paragraph “uniform continuity preserves uniform convergence…” should be helpful.
According to Remark 4.9, it seems that the proof strategy for polynomials can also be applied to nonpolynomials. This motivates a natural question: why can’t you apply the proof strategy for nonpolynomials to polynomials? I believe you can’t because Proposition 4.3 relies on the existing universal approximation results which requires \rho to be nonpolynomial (which is not mentioned in the main text). In my opinion, adding some comments on “why nonpoly techniques can’t be applied to poly activations” would help readers better understand the proof techniques.
Another question just out of curiosity: can you prove tightness of your construction, e.g., show that a square model with width n+m is NOT a universal approximator?