Sparse Linear Programming via Primal and Dual Augmented Coordinate Descent
Over the past decades, Linear Programming (LP) has been widely used in different areas and considered as one of the mature technologies in numerical optimization. However, the complexity offered by state-of-the-art algorithms (i.e. interior-point method and primal, dual simplex methods) is still unsatisfactory for problems in machine learning with huge number of variables and constraints. In this paper, we investigate a general LP algorithm based on the combination of Augmented Lagrangian and Coordinate Descent (AL-CD), giving an iteration complexity of O((log(1/ ))
2 ) with O(nnz(A)) cost per iteration, where nnz(A) is the number of non-zeros in the m × n constraint matrix A, and in practice, one can further reduce cost per iteration to the order of non-zeros in columns (rows) corresponding to the active primal (dual) variables through an active-set strategy. The algorithm thus yields a tractable alternative to standard LP methods for large-scale problems of sparse solutions and nnz(A) mn. We conduct experiments on large-scale LP instances from 1 -regularized multi-class SVM, Sparse Inverse Covariance Estimation, and Nonnegative Matrix Factorization, where the proposed approach finds solutions of 10 −3 precision orders of magnitude faster than state-of-the-art implementations of interior-point and simplex methods.
In this paper, the authors propose a method which combines augmented Lagrangian and coordinate descent techniques for solving large scale linear programs with a sparse constraint matrix.Pros and cons --------------+ Overall, this is a strong contribution. In the introduction, the authors did a good job motivating their paper and covering the related work. Coordinate descent is a good fit for solving the inner problem. + The authors give iteration complexity guarantees for the entire algorithm, even if the inner problem is solved inexactly.+ I particularly appreciated that the experiments cover three representative linear programs in machine learning: l1-regularized multiclass SVM, NMF and sparse covariance matrix estimation.- The projected Newton-CG (PNCG) method is not covered in the experiments.- The experiments are rather short and only indicate timing results. I would have preferred to see more experiments and move the PNCG part to the supplementary material.- There is no discussion whatsoever on how the parameter \eta_t was chosen. If the paper is accepted, it is very important to address this issue in the camera-ready version.- Same problem with regularization parameters in the experiments.- While much better than interior point and simplex methods, the timing results remain too large for practical use. In practice, one can add a little bit of l2 regularization to the problem (i.e., replace l1 regularization by elastic-net).Then the problem can be solved by modern solvers such as SDCA, which should be orders of magnitude faster than results shown in this paper.Detailed comments ------------------* The authors claim in several places in the paper that the conditioning of the problem doesn't worsen with the number of iterations. It would be interesting to briefly explain why.* line 101: non-negative constraint -> non-negativity constraint* line 176: g(y) is already defined in (2). Here you are overloading the definition so that g(y) becomes infinity if y is infeasible. I hadn't realized this at first glance and so (5) made no sense. Instead of overloading g(y), you could define a new function \tilde{g}(y) to make it more explicit.* problem (5): you didn't define \eta_t* line 230: you already defined \hat{F}(x) in (9)* line 236: add citation for generalized Hessian* line 259: clever -> efficient* Table 1-3: Barrier = IPM?