Parallel Successive Convex Approximation for Nonsmooth Nonconvex Optimization
Consider the problem of minimizing the sum of a smooth (possibly non-convex) and a convex (possibly nonsmooth) function involving a large number of variables. A popular approach to solve this problem is the block coordinate descent (BCD) method whereby at each iteration only one variable block is updated while the remaining variables are held fixed. With the recent advances in the developments of the multi-core parallel processing technology, it is desirable to parallelize the BCD method by allowing multiple blocks to be updated simultaneously at each iteration of the algorithm. In this work, we propose an inexact parallel BCD approach where at each iteration, a subset of the variables is updated in parallel by minimizing convex approximations of the original objective function. We investigate the convergence of this parallel BCD method for both randomized and cyclic variable selection rules. We analyze the asymptotic and non-asymptotic convergence behavior of the algorithm for both convex and non-convex objective functions. The numerical experiments suggest that for a special case of Lasso minimization problem, the cyclic block selection rule can outperform the randomized rule.
***** Update after author feedback ******I agree with the authors' response and trust them to make the suggested changes that will improve the paper. On the other hand, as this added discussion won't be double checked by reviewers, I strongly encourage them to have the modification reviewed by a critical colleague in the field to ensure its quality.*******************quality:  7 (out of 10)clarity: 8originality: 6significance: 7SUMMARY: The authors propose a simple parallel scheme for the block-coordinate optimization of the sum of a smooth (possibly non-convex) function with a non-smooth convex block-separable function. The scheme finds a target point for each block in parallel in a chosen subset of blocks by minimizing the sum of a strongly convex approximation to the smooth part on this block (with matching gradients) and the non-smooth part. Each block in this subset is then updated (in parallel) as a convex combination of the previous value and the target points. A parallel proximal gradient scheme can be obtained as a special case; though using a convex combination of the iterates yield a slightly different scheme than previous work. The suggested algorithm is very similar to [9], except that in [9] the subset was chosen using a greedy scheme (which can be expensive), whereas this submission explores both randomized schemes or a cyclic scheme. For these, the authors prove the asymptotic convergence to a stationary point of the algorithm under standard Lipschitz gradient conditions. They also prove a O(1/k) convergence rate (where k is the number of cycles for the cyclic scheme) to the global minimum for the convex case or for the square of the norm of the objective proximal gradient  for the non-convex case, for the algorithm with a constant step-size. They provide a succinct experimental comparison for a Lasso problem, observing that the cyclic scheme performs better than the randomized one, and that a speed-up can be obtained through parallelization (up to a point where the communication overhead starts to dominate).EVALUATION:I think this paper makes an interesting contribution to NIPS. There has been a lot of interest in parallel optimization schemes lately, and the one proposed in this paper, by analyzing general convex approximations to non-convex smooth objectives, is fairly general. Pieces of this submission has all been done somewhere else, but the combination is novel. For example, successive convex approximation schemes for non-convex problems was already analyzed in [24], but not for parallel updates. The contributions over [9] is to analyze the cyclic and randomized subset selection scheme, as well as to provide global convergence bounds (rather than just asymptotic convergence as in [9]). A contribution over several other parallel block coordinate  schemes which was proposed recently is to consider the general successive convex approximation approach (instead of standard proximal gradient updates), which can give tighter bounds.The paper is easy to follow, and the proofs are fairly self-contained. I was disappointed though that the authors didn't provide the proof for the cyclic updates for Lemma 3 (line 313-314) in supplementary material.The numerical experiments are mainly a sanity check, and don't necessarily provide that much insights. The proof of asymptotic convergence (p. 5 mainly, as p.4 is used in other proofs) don't provide much insights and is fairly technical -- I would have suggested to put this part in an appendix in the supplementary material, and use the extra page freed to actually discuss more the meaning of their convergence rate results and how they compare with all the related schemes proposed in the literature. For example, how do the constants compare; what kind of speed-up is expected for different standard scenarios; what insights do the convergence analysis provide for practical usage of their algorithm (especially for a NIPS audience); etc. I think this would make the paper more useful to the NIPS community.Pros:- Simple and general parallel optimization scheme for smooth (possibly non-convex) objective with non-smooth separable part. The generality of the setup could be useful for the types of problems considered by the NIPS community.- Fairly self-contained (and easy to read) convergence analysis for both convex and non-convex case, including both cyclic and randomized subset schemes.Cons:- No discussion providing insights about their convergence results or how they compare with other results in the literature.- Toy empirical comparison.== Other comments ==Typos:- Line 167, \tau_min is not defined.- Line 253: the limit should be removed in this sentence, i.e. "... we have h(...) < = h( ...) for all x_i ..., *for all j*.