Lipschitz regularity of deep neural networks: analysis and efficient estimation
Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-ofart methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks. Our experiments show that SeqLip can significantly improve on the existing upper bounds. Finally, we provide an implementation of AutoLip in the PyTorch environment that may be used to better estimate the robustness of a given neural network to small perturbations or regularize it using more precise Lipschitz estimations.
The paper studies the problem of estimating the Lipschitz constant of a mapping realized by a trained neural network.  Several distinct contributions are made:   (i) It is shown that exactly computing the Lipschitz constant of a two layer MLP (Multi Layer Preceptron) with ReLU activation is NP-hard (worst-case result).   (ii) A straightforward algorithm for bounding the Lipschitz constant of a computational auto-differentiable graph is given.  The algorithm, named AutoLip, is a generalization of multiplying spectral norms of Jacobians for a sequence of mappings, and hence usually provides pessimistic bounds.   (iii) A technique for efficiently applying power method to a high-dimensional affine transformation is given.  This technique assumes efficient implementation of the mapping in an auto-differentiable graph.  It allows estimating the leading singular value of the transformation, which is equal to the Lipschitz constant.   (iv) For the case of MLP, an algorithm named SeqLip is presented, which is less naive, and tighter than AutoLip.  The degree to which it is tighter is theoretically analyzed under simplifying assumptions. The paper concludes with experiments comparing different estimations of the Lipschitz constant for several trained networks.  In my opinion, the problem studied in this paper is important, not only for robustness to input perturbations (motivation given in the text), but also for improving recent generalization bounds that depend on Lipschitz constant.  I advise the authors to mention this latter point in the text.    In terms of quality, the paper is well written, and its contributions are interesting.  I did not verify the proofs in supplementary material, but the results seem very reasonable and I believe they are correct.  One point I find troubling is the connection, or lack thereof, to prior work -- absolutely nothing is said about existing literature.  I kindly ask the authors to include such reference in their rebuttal, and subsequently add it to the manuscript.  To my knowledge contributions (i)-(iv) above are novel.  Assuming this is the case, I recommend accepting the paper.  Points which I believe can improve this work:   * The text covers multiple ideas, many of which are in my opinion distinct, and vary in their importance.  Of contributions (i)-(iv) above, I think (iv) is the most interesting one, and (iii), which is also interesting, can be presented as part of (iv).   (i) and (ii) are less interesting (the first because it is a worst case analysis; the second because it is trivial).  There is also some text beyond (i)-(iv) which I think adds almost nothing -- the discussion on Lipschitz constants of activation, pooling, dropout and batch norm layers.  I believe the text would benefit from being more focused on the important parts.   * It seems that the improvement of SeqLip over AutoLip is not that significant (one order of magnitude, while AutoLip is likely overpessimistic by much more than that).  I think further refinement of SeqLip would be of great interest, and encourage the authors to pursue this goal, either in this work or in a follow-up.   * The experiments are very minimal, which is a shame because they are insightful.  I would add many more models to the evaluation, placing the results in supplementary material if needed.   * I think there are some typos in the text (for example in Eq 11).  This is not critical, because the reader can easily reproduce the derivations independently.  ---- Edit following discussion period: I have read the authors' response, and in particular their reference to existing literature.  I have also read the other reviews.  While I agree with the criticism of R3 with regards to the experiments, in my opinion, the contribution of this work is theoretical -- it presents a technique (SeqLip) for bounding the Lipschitz constant of a MLP, going a step beyond the trivial routine of multiplying spectral norms.  My stand on the paper remains -- I recommend its acceptance.