In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can define a smoothed version of the normalized margin which increases over time. We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem. Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks. We conduct several experiments to justify our theoretical finding on MNIST and CIFAR-10 datasets. Finally, as margin is closely related to robustness, we discuss potential benefits of training longer for improving the robustness of the model.
The goal of the paper is to formally prove that gradient flow / gradient descent performed on homogeneous neural network models maximizes the margin of the learnt function; assuming gradient flow/descent manages to separate the data. This is proved in two steps:
  1. Assuming that gradient descent manages to find a set of network parameters that separate the data, thereafter gradient flow/descent monotonically increases the normalized margin (rather an approximation of it).
  2. The limit points of optimization are KKT points of the margin maximization optimization problem.
While the main body of the paper presents a restricted set of results, the appendix generalizes this much further applying it to various kinds of loss functions (logistic/cross-entropy, exponential), to multi-class classification and to multi-homogeneous models. There seem to be many subtleties in the proofs and the paper seems to be quite thorough. (I must say that I'm not expert enough to assess the technical novelty of this paper over prior works.)
Recommendation:
I recommend "acceptance". The paper takes a significant step by unifying existing results on margin maximization and going beyond them. 
Technical comments:
- It is clear that in order to define margin meaningfully, some form of normalization is necessary. But a priori, $\|\theta\|_2^L$ is not the *only* choice; $\|\theta\|^L$ could also work for any norm $\|\cdot\|$. But perhaps the choice of $\|\cdot\|_2$ is special (as Thm 4.4 suggests). It will be nice to have some insights/comments on why this choice of $\|\cdot\|_2$ based normalization is the right one.
- The paper argues that having a larger margin helps in obtaining better robustness to adversarial perturbations (within $\|\cdot\|$ balls for some choice of $\|\cdot\|$). However note that the notion of "margin" is not just a function of the decision boundary, but instead depends on the specific function computed by the neural network --- this is unlike margin maximization in linear models, where "margin" in determined entirely by the decision boundary. As the paper argues, if we have an upper bound on the Lipschitz constant w.r.t. $\|\cdot\|$ norm, then we get a lower bound on required adversarial perturbations for any training point. However, this does not mean that training longer is necessarily better because by doing so, we might end up with a larger Lipschitz constant (even after normalizing). So even if the "margin" is larger, the actual adversarial perturbations (in $\|\cdot\|$ norm) allowed might get smaller. So I'm not sure how relevant this result is for adversarial robustness.