Efficient Structured Matrix Rank Minimization
We study the problem of finding structured low-rank matrices using nuclear norm regularization where the structure is encoded by a linear map. In contrast to most known approaches for linearly structured rank minimization, we do not (a) use the full SVD; nor (b) resort to augmented Lagrangian techniques; nor (c) solve linear systems per iteration. Instead, we formulate the problem differently so that it is amenable to a generalized conditional gradient method, which results in a practical improvement with low per iteration computational cost. Numerical results show that our approach significantly outperforms state-of-the-art competitors in terms of running time, while effectively recovering low rank solutions in stochastic system realization and spectral compressed sensing problems.
** I have read the authors’ response and they have provided satisfactory comments addressing my major concerns. I discuss some of these below (denoting added comments with **). I have read the other reviews and my rating of the paper hasn’t changed; I think it is a strong paper and should be accepted. I do not feel that the theoretical or experimental contribution pushes it into the oral/spotlight category.**The paper addresses the problem of finding structured low rank matrices for the case when the structure is encoded using a linear map and nuclear norm regularization is employed. The authors propose a modified formulation of the problem which makes it amenable to application of the generalized conditional gradient method. They provide an algorithm that incorporates other methods that accelerate convergence and show that it converges. Empirical studies suggest that the algorithm is significantly faster than its competitors and can scale to much larger problems. Strengths:(1) The paper addresses an optimization problem that lies at the core of multiple machine learning and signal processing tasks and the authors provide a novel procedure that can identify a suitable solution much faster than state-of-the-art approaches. The empirical studies demonstrate the efficacy of the approach for two useful application problems. Weaknesses:(1) The authors achieve the significant reduction in computational cost by modifying the objective function so that it is no longer directly penalizes the rank of the structured matrix. The authors make three reasonable arguments for doing so shortly after stating the objective function in (7).  The first two argue that the linear constraint could be satisfied asymptotically through a homotopy scheme or exactly through a projection, but the paper doesn’t explore either of these options, either empirically or theoretically. The theoretical result really addresses the convergence of the algorithm in terms of minimizing the modified objective function; it doesn’t provide a sense of the discrepancy between the identified solution and the more natural original formulation. Graphs 2(c) and 2(f) show some slightly strange behaviour in the rank of the solution as it converges. This behaviour is not discussed at all and it is not clear when the algorithm should be terminated to identify an appropriate solution.  ** The authors have argued that “any formulation that achieves this goal should be sufficient for the application”. I think this is a reasonable argument, but at the moment the paper is presented as though (7) is a relaxation/approximation of (5) – and is effectively trying to find the same solution. I think it would be useful to add a sentence or two making it clear that this is an alternative utility function that targets the same overall objective (low-rank model). **** With regard to the experimental results and the behaviour of the algorithm, it would be good to add some extra discussion either in the main paper or in the supplementary material expanding on what the authors have provided in their response” **Comments:Why does Theorem 1 cite [28]? Although there is a similar theorem in [28], with a similar proof approach, this doesn’t seem to be taken directly from there? What does ([28]) mean?** The authors acknowledge that the citation should appear in the proof, after a statement that the proof technique is inspired by the method in [28]. **