The Expressive Power of Neural Networks: A View from the Width
The expressive power of neural networks is important for understanding deep learning. Most existing works consider this problem from the view of the depth of a network. In this paper, we study how width affects the expressiveness of neural networks. Classical results state that depth-bounded (e.g. depth-2) networks with suitable activation functions are universal approximators. We show a universal approximation theorem for width-bounded ReLU networks: width-(n + 4) ReLU networks, where n is the input dimension, are universal approximators. Moreover, except for a measure zero set, all functions cannot be approximated by width-n ReLU networks, which exhibits a phase transition. Several recent works demonstrate the benefits of depth by proving the depth-efficiency of neural networks. That is, there are classes of deep networks which cannot be realized by any shallow network whose size is no more than an exponential bound. Here we pose the dual question on the width-efficiency of ReLU networks: Are there wide networks that cannot be realized by narrow networks whose size is not substantially larger? We show that there exist classes of wide networks which cannot be realized by any narrow network whose depth is no more than a polynomial bound. On the other hand, we demonstrate by extensive experiments that narrow networks whose size exceed the polynomial bound by a constant factor can approximate wide and shallow network with high accuracy. Our results provide more comprehensive evidence that depth may be more effective than width for the expressiveness of ReLU networks.
SUMMARY 
* This paper studies how width affects the expressive power of neural networks. It suggests an alternative line of argumentation (representation of shallow by deep narrow) to the view that depth is more effective than width in terms of expressivity. The paper offers partial theoretical results and experiments in support of this idea. 
CLARITY 
* The paper expresses the objectives clearly and is easy to follow. However, the technical quality needs attention, including confused notions of continuity and norms. 
RELEVANCE 
* The paper discusses that too narrow networks cannot be universal approximators. This is not surprising (since the local linear map computed by a ReLU layer has rank at most equal to the number of active units). Nonetheless, it highlights the fact that not only depth but also width is important. 
* The universal approximation results seem to add, but not that much to what was already known. Here it would help to comment on the differences. 
* The paper presents a polynomial lower bound on the required number of narrow layers in order to represent a shallow net, but a corresponding upper bound is not provided. 
* As an argument in favor of depth, the theoretical results do not seem to be conclusive. For comparison, other works focusing on depth efficiency show that exponentially larger shallow nets are needed to express deep nets. The paper also presents experiments indicating that a polynomial number of layers indeed allows deep narrow nets to express functions computable by a shallow net. 
OTHER COMMENTS 
* The paper claims that ``as pointed out in [2]: There is always a positive measure of network parameters such that deep nets can be realized by shallow ones without substantially larger size''.  This needs further clarification, as that paper states `` we prove that besides a negligible set, all functions that can be implemented by a deep network of polynomial size, require exponential size in order to be realized (or even approximated) by a shallow network.'' 
* In line 84, what is the number of input units / how do the number of parameters of the two networks compare? 
* In related works, how to the bounds from [14,18] compare with the new results presented in this paper? 
* In line 133. Lebesgue measurable functions are not generalizations of continuous functions. 
* In line 134 the argumentation for L1 makes no sense. 
* In line 163 ``if we ignore the size of the network ... efficient for universal approximation'' makes no sense. 
* The comparison with previous results could be made more clear. 
* Theorem 2 is measuring the error over all of R^n, which does not seem reasonable for a no-representability statement. 
* Theorem 3 talks about the existence of an error epsilon, which does not seem reasonable for a no-representability statement. Same in Theorem 4.  
* In line 235 the number of points should increase with the power of the dimension. 
CONCLUSION 
The paper discusses an interesting question, but does not answer it conclusively. Some technical details need attention.