Stochastic Spectral and Conjugate Descent Methods
The state-of-the-art methods for solving optimization problems in big dimensions are variants of randomized coordinate descent (RCD). In this paper we introduce a fundamentally new type of acceleration strategy for RCD based on the augmentation of the set of coordinate directions by a few spectral or conjugate directions. As we increase the number of extra directions to be sampled from, the rate of the method improves, and interpolates between the linear rate of RCD and a linear rate independent of the condition number. We develop and analyze also inexact variants of these methods where the spectral and conjugate directions are allowed to be approximate only. We motivate the above development by proving several negative results which highlight the limitations of RCD with importance sampling.
This paper focuses on quadratic minimization and proposes a novel stochastic gradient descent algorithm by interpolating randomized coordinate descent (RCD) and stochastic spectral descent (SSD). Those two algorithms were studied in prior but they have drawbacks to apply for practical usage. Hence, authors combine them by random coin-tossing (i.e., with probability \alpha run RCD and with probability 1-\alpha run SSD) and provide the optimal parameters to achieve the best convergence rate. In addition, they study the rate of their algorithm for some special cases, e.g., a matrix in objective has clustered or exponentially decaying eigenvalues. In experiments, they report practical behavior of their algorithm under synthetic settings.  The proposed algorithm takes advantages of two prior works, i.e., efficient gradient update from RCD and fast convergence from SSD. And it is interesting that the rate of the algorithm is between that of RCD and SSD, but not surprising. Moreover, authors study some extreme cases for RCD that the rate is arbitrary large and it is questionable that the proposed algorithm have the cases. It would be better to mention some extreme cases for SSCD.  And authors benchmark their algorithm under only synthetic dataset. It is desired to test the algorithm under real-world dataset and report its performance. Furthermore, there have been studied many algorithms for quadratic minimization, the proposed algorithm should compete with other works.  Overall, the main contribution of this paper is interesting with an intuitive interpolating approach and some case studies. However, the limited experiments make this work weaken. I consider this works as marginally above the acceptance.   ***** I have read the author feedback. I appreciate authors for providing the details about my questions. *****