ABSTRACT Deep (neural)  networks have been applied productively in a wide range of supervised and unsupervised learning tasks. Unlike classical machine learning algorithms, deep networks typically operate in the overparameterized regime, where the number of parameters is larger than the number of training data points. Consequently, understanding the generalization properties and the role of (explicit or implicit) regularization in these networks is of great importance. In this work, we explore how the oft-used heuristic of data augmentation imposes an implicit regularization penalty of a novel measure of the rugosity or "roughness" based on the tangent Hessian of the function fit to the training data.
This paper shows that a penalty term called rugosity captures the implicit regularization effect of deep neural networks with ReLU (and piecewise affine in general) activation. Roughly, rugosity measures how far the function parametrized as a deep network deviates from a locally linear function. 
The paper starts by showing that the amount of training loss increased from adding data augmentation is upper bounded in terms of (roughly) a Monte Carlo approximate to a Hessian based measure of rugosity. It then formally derives this measure of rugosity for networks with continuous piecewise affine activations. Finally, experimental evaluation for classification tasks on MNIST, SVHN and CIFAR shows that data augmentation indeed reduces the rogusity by a significant amount particularly when using the ResNet structure. A somehow surprising message is, however, that if one imposes explicit regularization with rugosity in lieu of data augmentation, then the better generalization usually seen from data augmentation no longer presents, though one does get a network with smaller rugosity.
Comments:
It is quite interesting to see that the rugosity measure proposed in the paper captures at least some aspects of the implicit regularization effect of data augmentation both in terms of theory (i.e. Theorem 1) and practical observations. My feeling is that rugosity is mostly a measure of the smoothness of the function parametrized by the neural network. From that perspective, how is the rugosity as a smoothness measurement for neural networks with piecewise affine activations different from the Lipschitz constant for general neural networks? My guess is that data augmentation also decreases the Lipschitz constant of a neural network near the training data points, but regardless of whether this is true or not, it is not clear if and how rugosity is better than Lipschitz constant for characterizing the implicit regularization of data augmentation. 
In addition, there have been many recent studies on showing that gradient penalty / Lipschitz regularization are useful for achieving better generalization and adversarial robustness, see e.g. [a,b,c]. The results in this paper on showing that regularizing rugosity does not improve accuracy seem to contradict with the conclusion of these prior studies. It is unclear to me whether this is caused by insufficient experimentation or if there is any fundamental difference between rugosity and Lipschitz regularization that I am missing.
[a] Finlay et al., Lipschitz regularized deep neural networks generalize and are adversarially robust
[b] Gouk et al., Regularisation of Neural Networks by Enforcing Lipschitz Continuity
[c] Thanh-Tung et al., Improving generalization and stability of GANs