Faster Projection-free Convex Optimization over the Spectrahedron
Minimizing a convex function over the spectrahedron, i.e., the set of all d ⇥ d positive semidefinite matrices with unit trace, is an important optimization task with many applications in optimization, machine learning, and signal processing. It is also notoriously difficult to solve in large-scale since standard techniques require to compute expensive matrix decompositions. An alternative is the conditional gradient method (aka Frank-Wolfe algorithm) that regained much interest in recent years, mostly due to its application to this specific setting. The key benefit of the CG method is that it avoids expensive matrix decompositions all together, and simply requires a single eigenvector computation per iteration, which is much more efficient. On the downside, the CG method, in general, converges with an inferior rate. The error for minimizing a -smooth function after t iterations scales like /t. This rate does not improve even if the function is also strongly convex. In this work we present a modification of the CG method tailored for the spectrahedron. The per-iteration complexity of the method is essentially identical to that of the standard CG method: only a single eigenvector computation is required. For minimizing an ↵-strongly convex and -smooth function, the expected error of the method after t iterations is:
) are the rank of the optimal solution and smallest nonzero eigenvalue, respectively. Beyond the significant improvement in convergence rate, it also follows that when the optimum is low-rank, our method provides better accuracy-rank tradeoff than the standard CG method. To the best of our knowledge, this is the first result that attains provably faster convergence rates for a CG variant for optimization over the spectrahedron. We also present encouraging preliminary empirical results.
where rank(X ⇤ ), min (X ⇤ ) are the rank of the optimal solution and smallest nonzero eigenvalue, respectively. Beyond the significant improvement in convergence rate, it also follows that when the optimum is low-rank, our method provides better accuracy-rank tradeoff than the standard CG method. To the best of our knowledge, this is the first result that attains provably faster convergence rates for a CG variant for optimization over the spectrahedron. We also present encouraging preliminary empirical results. of a real symmetric matrix. Such a decomposition requires O(d 3 ) arithmetic operations for a d ⇥ d matrix and thus is prohibitive for high-dimensional problems. An alternative is to use first-order methods that do not require expensive decompositions, but rely only on computationally-cheap leading eigenvector computations. These methods are mostly based on the conditional gradient method, also known as the Frank-Wolfe algorithm [3,12], which is a generic method for constrained convex optimization given an oracle for minimizing linear functions over the feasible domain. Indeed, linear minimization over the spectrahedron amounts to a single leading eigenvector computation. While the CG method has been discovered already in the 1950's [3], it has regained much interest in recent years in the machine learning and optimization communities, in particular due to its applications to semidefinite optimization and convex optimization with a nuclear norm constraint / regularization 1 , e.g., [10,13,17,19,22,2,11]. This regained interest is not surprising: while a full eigen-decomposition for d ⇥ d matrix requires O(d 3 ) arithmetic operations, leading eigenvecor computations can be carried out, roughly speaking, in worst-case time that is only linear in the number of non-zeros in the input matrix multiplied by either ✏ 1 for the popular Power Method or by ✏ 1/2 for the more efficient Lanczos method, where ✏ is the target accuracy. These running times improve exponentially to only depend on log(1/✏) when the eigenvalues of the input matrix are well distributed [14]. Indeed, in several important machine learning applications, such as matrix completion, the CG method requires eigenvector computations of very sparse matrices [13]. Also, very recently, new eigenvector algorithms with significantly improved performance guarantees were introduced which are applicable for matrices with certain popular structure [5,8,20].
The main drawback of the CG method is that its convergence rate is, in general, inferior compared to projection-based gradient methods. The convergence rate for minimizing a smooth function, roughly speaking, scales only like 1/t. In particular, this rate does not improve even when the function is also strongly convex. On the other hand, the convergence rate of optimal projection-based methods, such as Nesterov's accelerated gradient method, scales like 1/t 2 for smooth functions, and can be improved exponentially to exp( ⇥(t)) when the objective is also strongly convex.
Very recently, several successful attempts were made to devise natural modifications of the CG method that retain the overall low per-iteration complexity, while enjoying provably faster convergence rates, usually under a strong-convexity assumption, or a slightly weaker one. These results exhibit provablyfaster rates for optimization over polyhedral sets [7,15] and strongly-convex sets [6], but do not apply to the spectrahedron. For the specific setting considered in this work, several heuristic improvements of the CG method were suggested which show promising empirical evidence, however, non of them provably improve over the rate of the standard CG method [19,17,4].
In this work we present a new non-trivial variant of the CG method, which, to the best of our knowledge, is the first to exhibit provably faster convergence rates for optimization over the spectrahedron, under standard smoothness and strong convexity assumptions. The per-iteration complexity of the method is essentially identical to that of the standard CG method, i.e., only a single leading eigenvector computation per iteration is required. Our method is tailored for optimization over the spectrahedron, and can be seen as a certain hybridization of the standard CG method and the projected gradient method. From a high-level view, we take advantage of the fact that solving a`2-regularized linear problem over the set of extreme points of the spectrahedron is equivalent to linear optimization over this set, i.e., amounts to a single eigenvector computation. We then show via a novel and non-trivial analysis, that includes new decomposition concepts for positive semidefinite matrices, that such an algorithmically-cheap regularization is sufficient, in presence of strong convexity, to derive faster convergence rates.
This paper provides a new method for strongly convex smooth minimization over the spectrahedron under Linear Minimization Oracle with provable rate of convergence. The expected error of this method in some regimes is better than for the existing conditional-gradient-type methods. The performance gain of the proposed approach in comparison to existing methods is demonstrated on a problem of matrix completion.
Initial review.  Technical quality:whether experimental methods are appropriate, proofs are sound, results are well analyzed. Besides the found errors in the proofs, I have the following comments which, I hope explain my evaluation. 1. The claim in the Lemma 1, that $b$'s form a distribution is not proved and doesn't hold. But I didn't find a place, where this claim is used in the sequel. 2. It is not clear why, for the numerical experiments, the authors choose a heuristic modification of their algorithm, despite that it doesn't have a proved convergence theorem. It would be interesting also to see how the provable method works. 3. Perhaps, the averaging over 5 independent experiments does not say much about the advantages of the new method.  Novelty/originality: in any aspect of the work, theory, algorithm, applications, experimental. I find the proposed approach new and interesting. It seems that it works at least in practice. The problem is the theoretical justification.  Potential impact or usefulness: could be societal, academic, or practical and should be lasting in time, affecting a large number of people and/or bridge the gap between multiple disciplines. Preliminary experiments show that this method could potentially solve practical problems faster. I also find the idea of rank-one regularization worth noting. The problem is the theoretical justification.  Clarity and presentation: explanations, language and grammar, figures, graphs, tables, proper references. My overall impression is quite positive. But several places were unclear for me and should be improved. 1. It would be nice to explain, why the last  identity in the first line of (6) holds.  2. It seems that in (6) the minimization is over the set of all unit vectors. But it is not stated there. 3. There are some misprints, e.g. on the line 179 "proprieties" should be "properties". 4. I could not understand, why in the long derivation between lines 346 and 347,  the first inequality holds. 5. It is not clear form the provided explanation, why the bound for the first term in the last inequality in (17) holds. 6. It is not clear, why the inequality on the line 410 holds. 7. It is not clear, why the inequality on the line 418 holds.     ================================================ Update after author feedback.  Technical quality: I still have some doubts about the numerical experiments. The authors choose a heuristic modification of their algorithm, despite that it doesn't have a proved convergence theorem. Their explanations about line-search modification seem to be reasonable. Still, in the experiment part, they use greedy choice of the rank-one matrix instead of random choice for the provable algorithm. It is not clear, why one should do that.  Novelty/originality:  I find the proposed approach new and interesting. It seems that it works at least in practice. The problem is the theoretical justification.  Potential impact or usefulness:  Preliminary experiments show that this method could potentially solve practical problems faster. I also find the idea of rank-one regularization worth noting. The problem is the theoretical justification.  Clarity and presentation:  Unfortunately, there were too many places in the text, which were unclear for me. In their feedback, the authors tried to explain them, but at least the explanations for items 4 and 5, are still not clear. Also I add the points, which I considered as errors in the first version of my review. I think they should also be clarified in the text. 1. It would be nice to explain in the text, why the last identity in the first line of (6) holds.  2. It seems that in (6) the minimization is over the set of all unit vectors. But it is not stated there. 3. There are some misprints, e.g. on the line 179 "proprieties" should be "properties". 4. I could not understand, why in the long derivation between lines 346 and 347, the first inequality holds. 5. It is not clear form the provided explanation, why the bound for the first term in the last inequality in (17) holds. 6. It would be nice to explain in the text, why the inequality on the line 410 holds. 7. It would be nice to explain in the text, why the inequality on the line 418 holds. 8. The claim in Lemma 1, that $b$'s form a distribution should be deleted. 9. I think, it still should be explained in the text, why ||A*B||_F < = ||A||_{spectral}*||B||_F and why ||P_{Y,\tau}||_{spectral} \leq 1. 10. I think, it should be explained in the text, how the third inequality in (19) follows from the optimality of $w^*$. 11. The estimate (25) should also be explained more explicitly. Unfortunately, the explanation in the author's feedback did not help me to understand, how it is proved.