Limitations on Variance-Reduction and Acceleration Schemes for Finite Sum Optimization
We study the conditions under which one is able to efficiently apply variancereduction and acceleration schemes on finite sum optimization problems. First, we show that, perhaps surprisingly, the finite sum structure by itself, is not sufficient for obtaining a complexity bound ofÕ((n + L/µ) ln(1/ )) for L-smooth and µ-strongly convex individual functions -one must also know which individual function is being referred to by the oracle at each iteration. Next, we show that for a broad class of first-order and coordinate-descent finite sum algorithms (including, e.g., SDCA, SVRG, SAG), it is not possible to get an 'accelerated' complexity bound ofÕ((n+ nL/µ) ln(1/ )), unless the strong convexity parameter is given explicitly. Lastly, we show that when this class of algorithms is used for minimizing L-smooth and convex finite sums, the iteration complexity is bounded from below by Ω(n + L/ ), assuming that (on average) the same update rule is used in any iteration, and Ω(n + nL/ ) otherwise.
This papers covers some theoretical limitations on variance-reduced stochastic algorithms
for finite sum problems (SAG, SDCA, SAGA, SVRG etc.). Such algorithms have had some important
impact in machine learning & convex optimization as they have been shown to reach a linear convergence
for strongly convex problems with lipschitz gradient (log term) while having a factor in $n + \kappa$ (and not as $n\kappa$).
A first result of this paper is that the sequential iterative algorithm needs to know
which individual function has been returned by the oracle to reach the linear convergence
rate. Typically SAG or SAGA need this knowledge to update the gradient memory buffer
so it is not terribly surprising that knowing the function index is necessary.
It is then argued that such variance-reduced methods cannot be used with data augmentation
during learning. This is only true if the data augmentation is stochastic and if there is
not a finite set of augmented samples.
A second result concerns the possibility to accelerate in the "Nesterov sense" such
variance-reduced algorithm (go from $\kappa$ to $\sqrt{\kappa}}$). It is shown that
"oblivious" algorithms whose update rules are (on average) fixed for any iteration
cannot be accelerated.
A practical consequence of the first result in 3.2 is that knowing the value of the
strong convexity parameter is necessary to obtain an accelerated convergence
when the algorithm is stationary.
Such oblivious algorithms are further analysed using
restarts which are well known techniques to alleviate the non-adaptive nature of
accelerated algorithms in the presence of unknown local conditionning.
The paper is overhaul well written with some pedagogy.
Minor:
CLI is first defined in Def 2 but is used before.
Typo: L224 First, let us we describe -> First, let us describe