Regularization is one of the crucial ingredients of deep learning, yet the term regularization has various definitions, and regularization methods are often studied separately from each other. In our work we present a novel, systematic, unifying taxonomy to categorize existing methods. We distinguish methods that affect data, network architectures, error terms, regularization terms, and optimization procedures. We identify the atomic building blocks of existing methods, and decouple the assumptions they enforce from the mathematical tools they rely on. We do not provide all details about the listed methods; instead, we present an overview of how the methods can be sorted into meaningful categories and sub-categories. This helps revealing links and fundamental similarities between them. Finally, we include practical recommendations both for users and for developers of new regularization methods.
This paper is unusual in that it is more of a review than contributing novel knowledge. It considers a taxonomy of all the ways that machine learning (mostly deep learning) methods can achieve a form of regularization. 
Unfortunately, it starts with a definition of regularization ('making the model generalize better') which I believe misses the point which was made in Goodfellow et al 2016 ('intend to improve test error but not necessarily training error'), i.e., that we would like to separate as much as possible the regularization effects from the optimization effect. Indeed, under the definition proposed here, any improvement in the optimizer could be considered like a regularizer, so long as we are not in the overfitting regime. That does not sound right to me.
There are several places where the authors make TOO STRONG STATEMENTS, taking for truth what are simply beliefs with no strong supporting evidence (at least published). This is not good for a review and when making recommendations.
The other weakness I estimate in this paper is that I did not get a sense that the taxonomy really helped us (me at least) to get insight into the different mentions being cited. Besides the obvious proposal to combine ideas to write new papers (but we did not need that paper to figure that out) I did not find much meat in the 'future directions' section.
However, I that except in a few places the understand of the field displayed by the authors is pretty good and, with correction, could serve as a useful reference for students of deep learning. The recommendations were reasonable although lacking empirical support (or pointers to the literature), so I would take them somewhat carefully, more as the current 'group think' than ground truth.
Finally, here a few minor points which could be fixed.
Eq. 1: in typical DL, minimization is approximate, not exact, so the proposed formalism does not reflect reality.
Eq. 4: in many cases, the noise is not added (e.g. dropout), so that should be clarified there.
page 3, first bullet of 'Effect on the data representation': not clear, may want to give translations as an example of such transformations,.
page 8, activation functions: the ReLU is actually older than the cited papers, it was used by computational neuroscientists a long time ago. Jarrett 2009 did not use the ReLU but an absolute-value rectifier and it was Glorot 2011 who showed that the ReLU really kicked ass for deeper networks. Nair 2010 used the ReLU in a very different context (RBMs), not really feedforward multi-layer networks where it shines now.
In that same section (and probably elsewhere) there are TOO STRONG STATEMENTS, e.g., the "facts" mentioned are not facts but merely folk belief, as far as I know, and I would like to see well-done supporting evidence before treating those as facts. Note that approximating the sigmoid precisely would require many ReLUs!
page 8: it is not clear how multi-task learning fits under the 'architecture' formalism provided at the beginning of section 4.
section 7 (page 10): there is earlier work on the connection between early stopping and L2 regularization, at least dating back to Ronan Collobert's PhD thesis (with neural nets), probably earlier for linear systems.