A Dual-Augmented Block Minimization Framework for Learning with Limited Memory
In past few years, several techniques have been proposed for training of linear Support Vector Machine (SVM) in limited-memory setting, where a dual blockcoordinate descent (dual-BCD) method was used to balance cost spent on I/O and computation. In this paper, we consider the more general setting of regularized Empirical Risk Minimization (ERM) when data cannot fit into memory. In particular, we generalize the existing block minimization framework based on strong duality and Augmented Lagrangian technique to achieve global convergence for general convex ERM. The block minimization framework is flexible in the sense that, given a solver working under sufficient memory, one can integrate it with the framework to obtain a solver globally convergent under limited-memory condition. We conduct experiments on L1-regularized classification and regression problems to corroborate our convergence theory and compare the proposed framework to algorithms adopted from online and distributed settings, which shows superiority of the proposed approach on data of size ten times larger than the memory capacity.
The main result is the linear convergence rate in terms of outer rounds of the method, assuming exact solvers of the (still large) ERM subproblems which fit in memory.The presentation of the paper could be improved significantly, but I think the contributions are still valuable, practical and interesting. Several of my comments below are suggestions for further improvements. UPDATE: Unfortunately the experiments lack a comparison to Quasi-Newton methods and d-GLMNET (i.e. the serialized variants of these). this issue was not answered in the author feedback, which questions parts of the value of the experiments. 1) Exact solution of the subproblems is assumed (See also Section 4.3.1), which is disappointing in view of practical applications. Even Theorem 2 only provides approximate solutions for the inner solvers. On the positive side, the experiments already work with approximate inner solvers.2) Choice of \eta_t in theory and experiments: In the paper, please make more clear that the algorithm depends on this step-size parameter. In experiments, how was this parameter selected? How does the performance depend on the parameter?3) Basic consensus-type ADMM variants which are well-studied for this type of problem. Their discussion (i.e. the summary) should be moved from the appendix to the main paper, and made more precise. Currently a good amount of space in the main paper is used by redundant formulas (e.g. useless dual, if working on the augmented primal, many repetitions from Section 3 in Section 4). It would be important to more clearly explain the difference in the updates proposed here as to the ADMM updates currently in the Appendix (Algorithm 3 and 4). I.e. write the actual ADMM updates and L_k for the case here, and compare to (15).4) Experiments:The paper unfortunately lacks an experimental comparison with other state of the art distributed l1-solvers (i.e. their serialized counterparts), such as d-GLMNET [A], Shotgun [B] and most importantly orthant-wise variants of Quasi-Newton methods [C] which are standard for l1-problems.The experiments are currently presented in a way that is far from reproducible, and we'd hope this will be improved. For example, the stopping criteria for the subproblem solvers, the \eta_k parameters, dependence on the regularization parameters etc is not given. Also, timing plots alone might not be very informative, but it might be important to discuss the IO load and number of passes through the data. In particular for online methods such as MD/SGD, they might suffer unfairly from inefficient random access, but could also be run in a memory-aware fashion. The tradeoff of IO vs computation could also be investigated, i.e. how much does one benefit from more local computation before swapping in a new data block.5) The discussion around lines 422 suggests that in contrast to MD, the approach here has linear convergence. This however is very oversimplifying, as the linear convergence is in number of outer rounds, each round consisting in an exact solution of a problem in millions of datapoints (same structure as the original problem). In contrast, the convergence rates of the online methods is in terms of individual points processed. The constants in the linear rate theory should be discussed, and could potentially be bounded in some relevant cases, e.g. for Lasso.6) The level of generality and the function classes assumed for the main optimization problems should be stated more precisly.7) *Dual* nature of the algorithm: Why do you call the algorithm 'dual-BCD' (motivated by [31]), if it is here applied to (7), which is the primal problem, with augmentation? This also relates to 'dual' in the title of the paper, as well as the title of Section 3, which in this view is confusing to start with the dual (4) if in the end the interest is (7). The reader will always be confused since all your proposed 'dual' methods are not solving (4) but (7). Also in line 247, calling it just BCD (or primal BCD or saddle-point BCD) would make it easier for the reader to understand which methods work on the dual, and which on the primal.8) Presentation The paper in some places seems to have been written in a rush, and the presentation and related work discussions could be greatly improved. In particular, the paper would benefit from a more concise discussion of related proximal gradient methods and their rates, as there is existing literature already even for the case of approximate proximal steps. 9) Why don't you generalize your result by allowing arbitrary solvers for the subproblem, why restricting to BCD (Theorem 2 etc), especially as this gets easily confused with outer BCD? In this respect, Theorem 3 is crucial for your contribution, while Theorem 2 is not.10) Minor: In Section 3, the authors extend the block-minimization framework of [31,2] from linear SVM to general ERM problems. However, this was already done by [D,E] for the ERM dual, though not including the l1 primal case. The transformation of those algorithms from the distributed setting to the 'limited memory' case here is straightforward, same as for ADMM and related methods.11) About easily adapting local solvers around line 280: Make precise the comment only covers first-order solvers, not solvers which potentially rely on more problem structure.12) Clarify how \mu_Bk is a variable in (7), i.e. if alpha_Bk is also an optimization variable. Same later in (15) etc.  minor - w^*() in line 3.2 of algorithm 2 is not clearly defined. -the readability of the whole paper and in particular the algorithms could easily be improved. 064: descnet 065,321: subsumes -> includes 176: define the function F, \bar F (objective in which problem?) 286: _in is not defined and properly related with S or Theorem 2. appendix: 581: appendix of lemma? 589: prposed. 2-norm messed up 630: define assumptions on \mu in Theorem 5 695: w and z are vectors in (38), not numbers, so need norms 784: note that 785: the analysis of *** additional references:[A] I Trofimov and A Genkin. Distributed Coordinate Descent for L1-regularized Logistic Regression. arXiv.org, 2014.[B] J K Bradley, A Kyrola, D Bickson, and C Guestrin. Parallel Coordinate Descent for L1-Regularized Loss Minimization. ICML, 2011.[C] G Andrew and J Gao. Scalable training of L1-regularized log-linear models. ICML, 2007.[D] T Yang. Trading Computation for Communication: Distributed Stochastic Dual Coordinate Ascent. NIPS, 2013.[E] M Jaggi et al. Communication-Efficient Distributed Dual Coordinate Ascent. NIPS, 2014.