Stochastic Composite Mirror Descent: Optimal Bounds with High Probabilities
We study stochastic composite mirror descent, a class of scalable algorithms able to exploit the geometry and composite structure of a problem. We consider both convex and strongly convex objectives with non-smooth loss functions, for each of which we establish high-probability convergence rates optimal up to a logarithmic factor. We apply the derived computational error bounds to study the generalization performance of multi-pass stochastic gradient descent (SGD) in a non-parametric setting. Our high-probability generalization bounds enjoy a logarithmical dependency on the number of passes provided that the step size sequence is square-summable, which improves the existing bounds in expectation with a polynomial dependency and therefore gives a strong justification on the ability of multi-pass SGD to overcome overfitting. Our analysis removes boundedness assumptions on subgradients often imposed in the literature. Numerical results are reported to support our theoretical findings.
The paper presents a high probability analysis of SGD for non-smooth losses, with weak assumptions on domain and gradients. It presents many different results, some of them are very interesting while some others are nice to have but not surprising at all.  In details:  - Section 3.1 presents high probability convergence rates for convex smooth functions. The core trick is to show that the iterates of SGD are (almost) bounded with high probability for square summable stepsizes. This result is not surprising at all: classic results on SGD already proved convergence with probability 1 even under weaker assumptions (i.e. non-convex functions with well-behaved gradients, see for example [6]) and through the same reasoning. The high probability bounds are nice to have, but nothing earth-shattering.  - Section 3.2 has similar results for strongly convex functions. These are even less suprising: changing the proof of [17] to make it work for smooth losses is a simple excercise that many of us have already done on their own. The high probability bounds for strongly objective functions are well-known, even in the composite mirror descent algorithm. Also, despite the claims in lines 274-275, removing the boundedness assumption on the gradients to obtain high probability bounds is trivial: It is enough to add a projection step into the ball where the optimal solution lives, that is always known using f(0, z_t)+r(0) and the strong convexity constant that you assumed to know. Indeed, this is how the Pegasos software was analyzed and implemented, [Shalev-Shwartz et al. ICML'07] (not cited). The projection step results in empirical faster convergence, better resistance to mispecified stepsizes, and even with better constants in the convergence rate. So, not using a projection step is actually a very bad idea. Overall, this is also a nice to have result, but not too strong.  - Section 4 presents the application of these results to prove generalization bounds for a multiple pass procedure that minimizes the unregularized empirical error. The obtained convergence rate matches the one obtained in Theorem 3 in [Orabona, NIPS'14] (not cited) with worst constants, yet here the authors don't assume bounded gradients and run multiple passes. I found this result very interesting, probably the most important one in the paper. As said, a way to achieve this rate without tuning stepsizes was known, but the fact that SGD achieves this rate after multiple passes, with a very weak dependency on the number of passes, is somehow suprising to me and it improves the result obtained with the stability analysis of SGD.  Overall, in my opinion the result presented in Section 4 warrants the publication of this paper. Also, it would be nice to see less overselling about the bounded gradients business.