title: The Implicit Bias of Gradient Descent on Separable Data
abstract: We show that gradient descent on an unregularized logistic regression
  problem, for almost all separable datasets, converges to the same direction as the max-margin solution. The result generalizes also to other monotone decreasing loss functions with an infimum at infinity, and we also discuss a multi-class generalizations to the cross entropy loss. Furthermore,
  we show this convergence is very slow, and only logarithmic in the
  convergence of the loss itself. This can help explain the benefit
  of continuing to optimize the logistic or cross-entropy loss even
  after the training error is zero and the training loss is extremely
  small, and, as we show, even if the validation loss increases. Our
  methodology can also aid in understanding implicit regularization
  in more complex models and with other optimization methods. 
authors: Daniel Soudry Elad Hoffer Mor Shpigel Nacson Nathan Srebro
hIndex: Nathan Srebro 63