HONOR: Hybrid Optimization for NOn-convex Regularized problems
Recent years have witnessed the superiority of non-convex sparse learning formulations over their convex counterparts in both theory and practice. However, due to the non-convexity and non-smoothness of the regularizer, how to efficiently solve the non-convex optimization problem for large-scale data is still quite challenging. In this paper, we propose an efficient Hybrid Optimization algorithm for NOn-convex Regularized problems (HONOR). Specifically, we develop a hybrid scheme which effectively integrates a Quasi-Newton (QN) step and a Gradient Descent (GD) step. Our contributions are as follows: (1) HONOR incorporates the second-order information to greatly speed up the convergence, while it avoids solving a regularized quadratic programming and only involves matrixvector multiplications without explicitly forming the inverse Hessian matrix. (2) We establish a rigorous convergence analysis for HONOR, which shows that convergence is guaranteed even for non-convex problems, while it is typically challenging to analyze the convergence for non-convex problems. (3) We conduct empirical studies on large-scale data sets and results demonstrate that HONOR converges significantly faster than state-of-the-art algorithms.
This paper considers efficient implementations of non-convex sparse learning formulations. Recent studies have shown that many convex sparse learning formulations are inferior to their non-convex counterparts in both theory and practice. However, it is still quite challenging to efficiently solve non-convex sparse optimization problems for large-scale data. This paper presents a novel algorithm HONOR which is applicable for a wide range of non-convex sparse learning formulations.   One of the key ideas in HONOR is to incorporate the second-order information to greatly speed up the convergence, while unlike most existing second-order methods it avoids solving a regularized quadratic programming and only involves matrix-vector multiplications without explicitly forming the inverse Hessian matrix. Thus, HONOR has a low computational complexity at each iteration and it is scalable to large-size problems. The convergence for non-convex problems is typically challenging to analyze and establish. One major contribution of this paper is to establish a rigorous convergence analysis for HONOR, which shows that convergence is guaranteed even for non-convex problems. The key to the convergence analysis is the hybrid optimization scheme which chooses either a Quasi-Newton step or a Gradient Descent step per iteration. The presented analysis is nontrivial. It will be good to include a high-level description of the intuition behind the proposed hybrid scheme.       The empirical evaluation presented in this paper is convincing. The authors evaluate the proposed algorithm using large-scale datasets which include up to millions of samples and features. Two of the datasets include over 20 million features. Such scale of datasets is needed to evaluate the behavior of the algorithms. Results demonstrate that HONOR converges significantly faster than state-of-the-art algorithms. Some guidance on the selection of \epsilon will be helpful. It is mentioned that HONOR may have the potential of escaping from high error plateau which often exists in high dimensional non-convex problems. It will be interesting to explore theoretical properties of the HONOR solutions. The proposed algorithm empirically converges very fast. Is there any guarantee on the (local) convergence rate of HONOR?Can the proposed algorithm be extended to other sparsity-inducing penalties such as group Lasso and fused Lasso?