Quartz: Randomized Dual Coordinate Ascent with Arbitrary Sampling
We study the problem of minimizing the average of a large number of smooth convex functions penalized with a strongly convex regularizer. We propose and analyze a novel primal-dual method (Quartz) which at every iteration samples and updates a random subset of the dual variables, chosen according to an arbitrary distribution. In contrast to typical analysis, we directly bound the decrease of the primal-dual error (in expectation), without the need to first analyze the dual error. Depending on the choice of the sampling, we obtain efficient serial and mini-batch variants of the method. In the serial case, our bounds match the best known bounds for SDCA (both with uniform and importance sampling). With standard mini-batching, our bounds predict initial data-independent speedup as well as additional data-driven speedup which depends on spectral and sparsity properties of the data.
The paper presents a variant of dual coordinate ascent for regularized empirical risk minimization with smooth convex loss functions and strongly convex regularizers. The authors derive an update scheme where in each iteration only a subset of dual coefficients are updated. The authors proof the convergence of their method for any sampling scheme producing those subsets. Moreover, they derive sampling schemes to speedup the convergence. In experiments, the authors verify their theoretical findings.The paper is well-written and scientific-sounding. The theoretical as well as the experimental results suggest that the proposed method outperforms existing algorithms when trained highly parallel, especially for very sparse data. My only concern is about the experiments which are not very conclusive: - combination of dataset, setting and method is not consistent - why not performing serial and mini-batch experiments for all 6 datasets? - why not reporting SPDC-roh=1 together with the other methods the serial setting? - likewise, why not reporting on ASDCA in the mini-batch setting? - no reporting of wall-clock time; and no reporting of predictive performance and/or values of the objective; it's hard to tell wether primal dual gaps of 1e-4 and 1e-10 make any difference in practise - no setting other than block size m (=1), loss (smoothed hinge), and regularizer (l2); consider m>1 where sparsity may significantly drop, Ridge regression where the Lipschitz constant is unbounded, etc. - only one large real-world dataset; consider rcv1 and/or imagenet - please detail on the choice of lambda. It has a significant impact on the convergence speed, but also on the predictive performance of the resulting model. For very large datasets, "optimal" lambda effectively approaches zero; which (practical) impact would this have on the convergence?Minor comments for the authors: - line 224: consider "the number of matrices A_i with any nonzero entry in row j" (it wasn't quit clear to me what "blocks" refers to) - Eq 17: colon - line 295: should be omega_j; consider adding max_j omega_j \leq n - "average sparsity", in line 295, is misleading; omega_j tilde is the "average number of non-zero entries for dimension j"; omega is an (absolute) measure of density rather than a (relative) measure of sparsity. Similarly, in Table 3, where 100% sparsity denotes a fully dense dataset. - titles in Figure 1 and, in particular, in Figure 2 are not very helpful; consider adding the 3 artificial datasets in Table 3; currently the sparsity (their key property) is only stated in the caption of Fig 2. - Figure 1: consider reordering the methods where both Quartz variants should be red - supp.: last line before Ch. 8 should be m > 1