We revisit the bias-variance tradeoff for neural networks in light of modern empirical findings. The traditional bias-variance tradeoff in machine learning suggests that as model complexity grows, variance increases. Classical bounds in statistical learning theory point to the number of parameters in a model as a measure of model complexity, which means the tradeoff would indicate that variance increases with the size of neural networks. However, we empirically find that variance due to training set sampling is roughly constant (with both width and depth) in practice. Variance caused by the non-convexity of the loss landscape is different. We find that it decreases with width and increases with depth, in our setting. We provide theoretical analysis, in a simplified setting inspired by linear models, that is consistent with our empirical findings for width. We view bias-variance as a useful lens to study generalization through and encourage further theoretical explanation from this perspective.
This paper suggests to rethink about the bias-variance tradeoff from statistical machine learning in the context of neural networks. Based on some empirical observations, the main claims in this work are that (1) it is not always the case that the variance will increase when we use bigger neural network models (particularly, by increasing the network width); (2) the variance should be decomposed into two parts: one part accounts for the variance caused by random initialization of network parameters/optimization and the other part is caused by "sampling of the training set".
For the first claim is based the empirical observation that increasing the number of hidden units did not cause the incrase of variance (as in figure 1). However, to my understanding, it only means increasing the number of hidden units is probably not a good way to increase the network capacity. In other words, this cannot be used as an evidence that the bias-variance tradeoff is not valid in neural network learning.
For the second claim, I don't like the way that they decompose the variance into two parts. To be clear, the classical bias-variance tradeoff doesn't consider the optimization error as an issue. For a more generic view of machine learning errors, please refer to "The Tradeoffs of Large Scale Learning" (Bottou and Bousquet, 2008). In addition, if the proposed framework wants to include the optimization error, it should also cover some other errors caused by optimization, for example, early stopping and the choice of a optimization algorithm.
Besides these high-level issues, I also found the technical parts of this paper is really hard to understand. For example,
- what is exactly the definition of $p(\theta|S)$? The closely related case I can think about is in the Baysian setting, where we want to give a prior distribution of model (parameter). But, clearly, this is not the case here. 
- similar question to the "frequentist risk", in the definition of frequentist risk, model parameter $\theta$ should be fixed and the only expectation we need to compute is over data $S$
- in Eq. (5), I think I need more technical detail to understand this decomposition.