This article conerns the expressive power of depth in neural nets with ReLU activations . We prove that ReLU nets with width 2d+2 can approximate any continuous scalar function on the d-dimensional cube [0,1]^d arbitrarily well. We obtain quantitative depth estimates for such approximations . Our approach is based on the observation that ReLU nets are particularly well-suited for representing convex functions. Indeed, we give a constructive proof that ReLU nets with width d+1 can approximate any continuous convex function of d  arbitrarily well. Moreover, when approximating convex, piecewise affine functions by width d+1 ReLU nets, we obtain matching upper and lower bounds on the required depth, proving that our construction is essentially optimal.  ]
This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width w_{\text{min that ReLU nets of width w_{\text{min can approximate any continuous function on the unit cube [0,1]^d aribitrarily well? For ReLU nets near this minimal width, what can one say about the depth necessary to approximate a given function? We obtain an essentially complete answer to these questions for convex functions . Our approach is based on the observation that , due to the convexity of the ReLU activation, ReLU nets are particularly well-suited for representing convex functions. In particular, we prove that ReLU nets with width d+1 can approximate any continuous convex function of d variables arbitrarily well. Moreover, when approximating convex, piecewise affine functions by such nets, we obtain matching upper and lower bounds on the required depth, proving that our construction is essentially optimal. These results then give quantitative depth estimates for the rate of approximation of any continuous scalar function on the d-dimensional cube 0,1]^d by ReLU nets with width d+3.
R
conerns
concerns
R
. We prove
and bounded width. We are particularly interested in the following questions: what is the minimal width w_{\text{min
R
with width 2d+2
of width w_{\text{min
D
scalar
R
d-dimensional
unit
R
arbitrarily well. We obtain quantitative depth estimates for such approximations
aribitrarily well? For ReLU nets near this minimal width, what can one say about the depth necessary to approximate a given function? We obtain an essentially complete answer to these questions for convex functions
R
ReLU
, due to the convexity of the ReLU activation, ReLU
R
Indeed, we give a constructive proof
In particular, we prove
A
variables
R
width d+1 ReLU
such
A
These results then give quantitative depth estimates for the rate of approximation of any continuous scalar function on the d-dimensional cube
A
0,1
A
^d by ReLU nets with width d+3.