Mixed Linear Regression with Multiple Components
In this paper, we study the mixed linear regression (MLR) problem, where the goal is to recover multiple underlying linear models from their unlabeled linear measurements. We propose a non-convex objective function which we show is locally strongly convex in the neighborhood of the ground truth. We use a tensor method for initialization so that the initial models are in the local strong convexity region. We then employ general convex optimization algorithms to minimize the objective function. To the best of our knowledge, our approach provides first exact recovery guarantees for the MLR problem with K 2 components. Moreover, our method has near-optimal computational complexity e O(Nd) as well as near-optimal sample complexity e O(d) for constant K. Furthermore, we show that our nonconvex formulation can be extended to solving the subspace clustering problem as well. In particular, when initialized within a small constant distance to the true subspaces, our method converges to the global optima (and recovers true subspaces) in time linear in the number of points. Furthermore, our empirical results indicate that even with random initialization, our approach converges to the global optima in linear time, providing speed-up of up to two orders of magnitude.
The paper under review studies the mixed linear regression problem, where the goal is to recover multiple underlying linear models from their unlabeled linear measurements.   The paper proposes a non-convex objective function, which, as shown in the paper, is close to being locally strongly convex in the neighborhood of the ground truth, and use a tensor method for initialization so that the initial modes actually fall into this strongly convex regime.   Specifically, the paper assumes pairs (y_i,x_i) to be generated as  y_i = < w_{z_i},x_i > ,   where z_i is an multinomial distributed index, w_k is the latent parameter of the k-th mixture component, and x_i is Gaussian.   Theorem 1 guarantees that provided the parameters w_k are independent of the samples and lie in the neighborhood of the solution, the Hessian is positive semi-definite (PSD) at the w_k with high probability. This does not imply, however, that the function is strongly convex around the true solution, as the statement is only for the Hessian at a set of given w_k (i.e., at one point, a set of w_k corresponds to one point in the parameter space).   Theorem 2 proves a similar statement than Theorem 2, this time, however, the w_k are not assumed to be independent of the samples. The cost for this stronger statement is that the w_k need to be significantly closer to the w_k^ast.  Again, the statement is for a given set of w_k, and does not hold for all w_k in a ball with high probability.   From those statements, the paper concludes that certain convex algorithms such as gradient descent converge to the true solution.  For this conclusion to be true, more has to be shown; I don't see how from  point wise strongly convexity, convergence follows (why don't we need strong convexity for all points in a ball w.h.p.?).   In section 3.3, the paper proposes a tensor method to initialize the parameters.  This initialization is interesting in that it provides an initialization that falls within the radius of point wise strong convexity of Theorem 1.  Additionally, it has complexity O(KdN + K^3 N poly(K) ). Thus, the complexity is small if K is small; however, for moderate to large K, the complexity is large.   In Section 4, the paper consider extensions to the subspace clustering problem.   Finally, the paper provides simulation results where the data points y_i are corrupted by additive noise. 
Overall I think this paper presents an interesting approach, well suited for NIPS. Below are some comments and and also concerns.   1. My strongest concern is on whether Theorem 1 and 2 indeed guarantee convergence of algorithms like gradient descent. Specifically, Theorem 1 and 2 show that with high probability, for a given point {w_k}_k sufficiently close to {w_k^ast}_k, the Hessian is positive semi definite (PSD)---as the paper puts it ``strongly convex at a point''. However the definition of a parameter converging algorithm (Def. 1) and the standard convergence proofs of gradient descent require the function to be strongly convex in a region, not only at a point.  Note that Theorem 1 and 2 do not guarantee that the Hessian is PSD at all points simultaneously.  Maybe Theorem 1 and 2 can be used together with an epsilon-net argument and appropriate union bounds to show that the function is strongly convex in a certain region around the optimum. However this is not done in the current version of the paper.  It is therefore not clear to me whether Corollary 1 and the conclusion in Section 3.1 hold.  Could the paper please address this point by providing a formal proof of Corollary 1 (i.e., of the statement of Theorem 1 implying convergence of say gradient descent)?    2. The proofs are not very formal. Specifically, Appendix A states several lemmas guaranteeing that certain events hold with high probability, and, e.g., the proof of Theorem 1 simply assumes some of those events throughout the proof. However, the proof does not explain why the assumptions of the lemmata hold, and does not put together the events via say the union bound. Additionally, in the derivations the paper does not explain where exactly the assumptions come in.  This makes it difficult and time consuming to verify the correctness of the proofs.   3. Overall the paper is well written; the proofs however miss proper punctuation throughout.   4. Theorem 3 makes some abstract assumptions on the original w_k^ast (Assumption 1). Could the paper give some intuition for which constellations of w_k those assumptions will be satisfied, and for which they fail? E.g., by discussing examples.   5. Theorem 5 on subspace clustering assumes that the subspaces are already initialized in a way so they are very, very close to the true solution (Eq. (9)) (exponentially small in K!). That seems to be a very stringent condition, as it is not clear how one finds such an initialization.   6. Subspace clustering: It is very interesting that no conditions on the affinity of the subspaces are needed. Is the intuition for this result, that the paper does not consider actual clustering, but subspace recovery? E.g., if two subspaces are exactly the same, this would not be an issue for subspace recovery, as considered in the paper, but would make the clustering problem ill posed as a given point could be assigned to one or the other subspace.   7. The numerical results show that the algorithm does very well in terms of running time, which is great. However, the results are for K=3. For larger K, one would expect the results to be different, specifically, the other algorithms would, for large enough K, perform better in terms of computation time than the proposed algorithm PSC. It would be interesting to show simulation for larger values of K, to see for which K PSC is faster, and for which it is not. This would also illustrate that PSC is not uniformly better.