MixLasso: Generalized Mixed Regression via Convex Atomic-Norm Regularization
We consider a generalization of mixed regression where the response is an additive combination of several mixture components. Standard mixed regression is a special case where each response is generated from exactly one component. Typical approaches to the mixture regression problem employ local search methods such as Expectation Maximization (EM) that are prone to spurious local optima. On the other hand, a number of recent theoretically-motivated Tensor-based methods either have high sample complexity, or require the knowledge of the input distribution, which is not available in most of practical situations. In this work, we study a novel convex estimator MixLasso for the estimation of generalized mixed regression, based on an atomic norm specifically constructed to regularize the number of mixture components. Our algorithm gives a risk bound that trades off between prediction accuracy and model sparsity without imposing stringent assumptions on the input/output distribution, and can be easily adapted to the case of non-linear functions. In our numerical experiments on mixtures of linear as well as nonlinear regressions, the proposed method yields high-quality solutions in a wider range of settings than existing approaches.
This paper considers a generalized version of the mixed regression problem, where we observe a collection of input-output samples, with the output corresponding to an additive combination of several mixture components/functions, and the goal is to find a collection of K functions that minimise the risk.   The corresponding ERM problem is NP-Hard to solve due to combinatorial constraints. The authors propose to relax these constraints by replacing them with an atomic norm regularizer they introduce as an "approximation" of the number of components. They propose to solve the resulting convex problem using a greedy algorithm.  Their analysis show that the solutions obtained by their approach achieve epsilon-optimal risk using a linear number of samples (both in terms of K and the dimension D) and O(K/epsilon) number of components, thus improving over the state-of-the-art in terms of number of sample complexity. The numerical experiments confirm this improvement.   The paper is written in a fairly clear way, and the approach of solving a convex relaxation of the mixed regression problem and the obtained sample complexity are compelling contributions. However, the introduced greedy algorithm (Algo. 1) seems unnecessarily computationally expensive!  In particular,  step 3 of  Algorithm 1 requires minimizing the loss function (7) over the current active set. The authors propose to solve this using proximal gradient algorithm (eq. 9), which requires solving a least squares problem at each iteration. Moreover, the convergence rate provided in Theorem 1 assumes that step (3) is solved exactly, which is not possible. Note also that the obtained convergence rate does not guarantee eps-optimal solution: F(c^T) - F(c^*) <= max{ O( ||c^*||_1 ), O(||c^*||^2_1 / T)}.   Why not solve instead the constrained formulation min g(M) s.t. ||M||_S <= K? This can simply be solved by Frank-Wolfe algorithm which only requires step 1 of Algorithm 1, and has a known convergence rate of O(1/T).   I think the paper could be improved by replacing Algorithm 1 with Frank-Wolfe and modifying the analysis in Theorem 2 accordingly.  Update: I thank the authors for their responses to my questions. Please note that Frank-Wolfe variants for the regularized form were also considered in the literature. For example, the paper "Conditional Gradient Algorithms for Norm-Regularized Smooth Convex Optimization" by Z. Harchaoui, A. Juditsky, and A. Nemirovski, provides a variant of Frank-Wolfe, which precisely address problems regularized with a norm, and is more computationally efficient than the proposed Algorithm 1. It is worth mentioning such algorithms in the paper, with an explanation (if any) on why Algorithm 1 might still be preferable in this case (for ex because it leads to easier analysis).