Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
We suggest a general oracle-based framework that captures different parallel stochastic optimization settings described by a dependency graph, and derive generic lower bounds in terms of this graph. We then use the framework and derive lower bounds for several specific parallel optimization settings, including delayed updates and parallel processing with intermittent communication. We highlight gaps between lower and upper bounds on the oracle complexity, and cases where the "natural" algorithms are not known to be optimal.
The paper presents lower bounds for parallel computation of population risk minimization over a ball using a DAG computation model, aiming to capture both computation and communication over parallel processors. Along the line of a series of papers [3,8,9,15,26,27] trying to provide lower bounds inspired lower bound of Nemirofski [19] on the serial setting, the present paper assumes queries are submitted to either a gradient or a prox oracle. Such queries constitute nodes in a graph, and the decision of which query to submit, as well as what value to "output" at a given node as a solution to the problem, depend only queries (and responses) observed *on the predecessors" of the nodes in the DAG. The authors provide lower bounds for the two oracles in this setting, depending on the Lipschitz constant and smoothness of the objective, as well as on the depth and size of the DAG and the radius of the constraint set. Going beyond these bounds, the authors discuss corresponding upper bounds for different topologies corresponding to different communication patterns, linking them to either existing algorithm and already derived upper bounds, or improving/tightening the gap with new proposed algorithms when the latter is not possible (though a few cases remain open).  The paper is timely, rigorous, and extremely well-written, and the results are of significant consequence. Though the analysis builds upon earlier work (mostly [3,9,26]), the authors amend several assumptions made in the prior art regarding the type of communication allowed. The assumptions considered here (that communication is restricted to sharing computational outcomes across the DAG) are both natural and more general than prior art.  The authors also identify  and elucidate aspects of their proof and function construction that differ from prior work in the end of Section 3 (thank you!).  There is no related work section, but the paper is well positioned with respect to related work spread out throughout the text. Section 4 in particular does a very good job linking lower bounds to a menagerie of existing/proposed algorithms, their upper bounds, and respective gaps (if they exist).  Though this section is lucid and readable, it would be good to be more precise and formal about how A-MB-SGD in [10,14], the algorithm in [11], or SVRG [13,15] indeed fall under the corresponding DAG models (and indeed resort to respective oracle queries) as the ones used in the paper. To deal with space constraints and maintain the readability of Section 4, this formal derivation can be added in the supplement.   Nits: Line 194 on Page 6, 11→ (11)  Line 203 on Page 6, j=1…\tau → j=1,...,\tau (missing commas) Line 241 on Page 7, approach Instead → approach. Instead Line 259 on Page 7, posses → possesses  Line 321 on Page 8, first order → first-order