The universal approximation theorem, in one of its most general versions, says that if we consider only continuous activation functions σ, then a standard feedforward neural network with one hidden layer is able to approximate any continuous multivariate function f to any given approximation threshold ε, if and only if σ is non-polynomial. In this paper, we give a direct algebraic proof of the theorem. Furthermore we shall explicitly quantify the number of hidden units required for approximation. Specifically, if X ⊆ R n is compact, then a neural network with n input units, m output units, and a single hidden layer with n+d d
hidden units (independent of m and ε), can uniformly approximate any polynomial function f : X → R m whose total degree is at most d for each of its m coordinate functions. In the general case that f is any continuous function, we show there exists some N ∈ O(ε −n ) (independent of m), such that N hidden units would suffice to approximate f . We also show that this uniform approximation property (UAP) still holds even under seemingly strong conditions imposed on the weights. We highlight several consequences: (i) For any δ > 0, the UAP still holds if we restrict all non-bias weights w in the last layer to satisfy |w| < δ. (ii) There exists some λ > 0 (depending only on f and σ), such that the UAP still holds if we restrict all non-bias weights w in the first layer to satisfy |w| > λ. (iii) If the non-bias weights in the first layer are fixed and randomly chosen from a suitable range, then the UAP holds with probability 1.
UPDATE TO MY EARLIER REVIEW
============================
Since this paper presets new findings that will be of significant interest to much of ICLR's audience, and the paper is is well-written, I am changing my rating to "Accept". Since Reviewer #1 did not submit a review and Reviewer #2 indicated that (s)he does not feel well-qualified to review this paper (it is very much on the theoretical side after all), it would be great to get one further review from an area chair or otherwise qualified person. 
MY EARLIER REVIEW
=================
This this exciting submission presents a new proof of Leshno's version of the universal approximation property (UAP) for neural networks  -- one of the foundational pillars of our understanding of neural networks. The new proof provides new insights into the universal approximation property. I consider these the main contribution of the paper. Specifically, the authors
- provide an upper bound on the required width for the neural network
- show that the approximation property still holds even if strong further requirements are imposed on the weights of the first or last layer. 
I rate this submission a weak accept. It’s a very good paper. The work makes useful contributions that should and will be of interest to many in the field. The paper is generally well-written. 
Some remarks:
- Being somewhat long, the “Proof of Theorem 3.1” would be a much better read if the authors prefixed it  with an outline of the strategy that the proof takes. 
- The authors point out that the lack of dependence of Theorem 3.1 on epsilon is surprising, and cite Lin’s work from 2017 who previously found such an independence. Lin’s derivation of the epsilon-independent UAP is much more intuitive than that of this submission, in which the epsilon independence really pops out somewhat magically and for me only made sense when I read the paper again. I would encourage the authors to add to Lin’s paper’s citation sentence that this paper motivates the epsilon independence well. Alternatively, the authors could add a few sentences to their paper to provide intuition on how the epsilon-independence comes about in their line of argument.