Stochastic Chebyshev Gradient Descent for Spectral Optimization
A large class of machine learning techniques requires the solution of optimization problems involving spectral functions of parametric matrices, e.g. log-determinant and nuclear norm. Unfortunately, computing the gradient of a spectral function is generally of cubic complexity, as such gradient descent methods are rather expensive for optimizing objectives involving the spectral function. Thus, one naturally turns to stochastic gradient methods in hope that they will provide a way to reduce or altogether avoid the computation of full gradients. However, here a new challenge appears: there is no straightforward way to compute unbiased stochastic gradients for spectral functions. In this paper, we develop unbiased stochastic gradients for spectral-sums, an important subclass of spectral functions. Our unbiased stochastic gradients are based on combining randomized trace estimators with stochastic truncation of the Chebyshev expansions. A careful design of the truncation distribution allows us to offer distributions that are varianceoptimal, which is crucial for fast and stable convergence of stochastic gradient methods. We further leverage our proposed stochastic gradients to devise stochastic methods for objective functions involving spectral-sums, and rigorously analyze their convergence rate. The utility of our methods is demonstrated in numerical experiments.
Spectral optimization is defined as finding $\theta$ that minimizes $F(A(\theta)) + g(\theta)$ where $A(\theta)$ is a symmetric matrix and $F$ typically the trace of an analytic function i.e. $F(A) = tr(p(A))$ where $p$ is a polynomial.   They propose an unbiased estimator of $F$ by randomly truncating the Chebyshev approximation to $F$ and doing importance sampling. Moreover, they calculate the optimal distribution for this importance sampling.  They demonstrate how this method would be used for SGD and stochastic Variance Reduced Gradient. Finally, they demonstrate two applications---matrix completion and minimizing log determinant for fitting Gaussian processes.  This paper is very clearly written, given the theoretical depth of the work. The main result is the optimal distribution for random truncation. The authors do an excellent job of defining and motivating this problem, proving their result, and demonstrating its effectiveness in two applications.  I would strongly argue for accepting this paper.  ## Questions ##  1. As pointed out in the paper, several other papers have taken similar approaches to approximating $tr(f(A))$ using Chebyshev approximations e.g. [6, 16, 28, 25]. However, I haven't seen a proof that importance sampling truncated series is strictly better than importance sampling individual terms. Have you been able to prove such a result? 2. In algorithm 1 and 2, it seems like the only stochasticity comes from sampling Rademacher vectors. We can imagine a case where $A(\theta)$ can only be calculated relative to a specific batch of data (e.g. the Hessian of a NN for a specific batch). Are you specifically not treating this case, or do you believe you have accounted for it here? 3. To make the two applications more explicit for people not very familiar with GPs or matrix completion, I would suggest writing out the objective that is being minimized in a form similar to $F(A(\theta)) + g(\theta)$ just to make the correspondence explicit.