ResNet with one-neuron hidden layers is a Universal Approximator
We demonstrate that a very deep ResNet with stacked modules that have one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. ℓ 1 (R d ). Due to the identity mapping inherent to ResNets, our network has alternating layers of dimension one and d. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension d [21,11]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.
In this paper, the authors demonstrated the universal approximation of ResNet with one-neuron hidden layer. In more details, ResNet with one single neural per hidden layer can approximate any Lebesgue-integrable functions to any accuracy as the depth goes to infinity. This also shows that ResNet has more representation power than narrow deep networks which are not universal approximators if the width is the input dimension. This result seems interesting since universal approximation is a fundamental property for any models. However, my comment may not be correct since I am not familiar with approximation theory of neural networks. I also did not check the involved proof.  The authors shows that the universal approximation of ResNet with one-neuron hidden layer implies the width-d+1 fully connected network are universal approximators. I did not fully understand this. From my understanding, it seems that ResNet with one-neuron hidden layers can not be understood as width-d+1 fully connected networks since there is no activation function applied to the identity map. Furthermore, there is a linear transformation applied to the activation function in the ResNet architecture.  After rebuttal: The authors' response addresses my concerns. I am in favor of its acceptance.