Recent theoretical work has demonstrated that deep neural networks have superior performance over shallow networks, but their training is more difficult, e.g., they suffer from the vanishing gradient problem. This problem can be typically resolved by the rectified linear unit (ReLU) activation. However, here we show that even for such activation, deep and narrow neural networks (NNs) will converge to erroneous mean or median states of the target function depending on the loss with high probability. Deep and narrow NNs are encountered in solving partial differential equations with high-order derivatives. We demonstrate this collapse of such NNs both numerically and theoretically, and provide estimates of the probability of collapse. We also construct a diagram of a safe region for designing NNs that avoid the collapse to erroneous states. Finally, we examine different ways of initialization and normalization that may avoid the collapse problem. Asymmetric initializations may reduce the probability of collapse but do not totally eliminate it.
In this paper, the authors investigate the collapse of deep and shallow network to a constant function under the following setting:
1. Very shallow networks, width ~ 10
2. ReLU activation function.  
3. Symmetric weights and biases initialization.  
4. Vanilla feed forward networks.  
The main take-home message is: don't use neural networks (NNs) that are both deep and shallow.  
The theoretical analysis is built on the observation: 
1. Every neuron (after applying ReLU) is equal to zero with probability 1/2. 
2. For narrow network and for any fixed input, there is a high chance that all neurons in a particular hidden layer are all zero. All neurons after that layer are all zero if zero-bias initialization is used.  
3. The authors conclude that derivatives of all parameters vanish but the bias of the last layer.
4. As a result, the network collapse to its mean (median) if mean squared loss (L1 loss) is used because only the bias of the last layer is being updated during training. 
Pros.
1. I think the phenomenon that shallow and deep NNs collapse to a constant is very interesting. 
2. The authors provide empirical and theoretical insights in favor of wider networks: have a better chance to avoid vanishing gradients.  
3. For shallow networks, it might be better not to use ReLU.  
Cons:
1.The analysis works in a very limited setting, works for ReLU but not other activations: tanh, erf, SELU etc. 
2. Very shallow networks are not popular in practice. Width>=100 is popular for fully-connected layers. 
3. The phenomenon observed by the paper can easily be addressed using any of the following trick: 
   3.1. Non-symmetric initialization  (set the mean to be non-zero). 
   3.2 . wider networks. 
4. Most of the analysis (and theorems)  are about one single input. In another word, distribution of the inputs have not been taken into account.  
5. I don't think the author provides a completely rigorous justification for the collapse phenomenon.  
Other comments. 
1. Eq (2) in page 4 is not trivially correct. The expectation operator (w.r.p. to lower layers) is moved into the activation function phi, justification is needed for this step.  
2. Theorem 4: when the Lebesgue measure of $\Omega$ is NOT finite, it is unclear how to define a uniform probability distribution on it.  
3. Theorem 4: the integrability assumption on y should depend on the loss: for L2 loss (L1 loss), squared (absolutely) integrable  should be used. They are not the same.