Efficient Globally Convergent Stochastic Optimization for Canonical Correlation Analysis
We study the stochastic optimization of canonical correlation analysis (CCA), whose objective is nonconvex and does not decouple over training samples. Although several stochastic gradient based optimization algorithms have been recently proposed to solve this problem, no global convergence guarantee was provided by any of them. Inspired by the alternating least squares/power iterations formulation of CCA, and the shift-and-invert preconditioning method for PCA, we propose two globally convergent meta-algorithms for CCA, both of which transform the original problem into sequences of least squares problems that need only be solved approximately. We instantiate the meta-algorithms with state-of-the-art SGD methods and obtain time complexities that significantly improve upon that of previous work. Experimental results demonstrate their superior performance.
The paper considers a stochastic version in CCA.  CCA is a useful tool that, given two snapshots of different set of variables (that come, however, from the same set of subjects), we are interested in revealing the most important correlations between the variables. State of the art algorithms work mostly in an offline fashion: given all datapoints from each set of variables, the task is to solve the maximization of a quadratic bilinear form, over quadratic constraints.  While the problem has a "closed-form" solution (after matrix transformations such as matrix inversions), only until recently some online/stochastic methods have been proposed. This paper considers such scenario for CCA and proposes 2 algorithms that solve the online CCA problem with global approximation guarantees.  The authors claim that this is the first time such guarantees are provided for such setting.  Experimental results show the superiority of the algorithms compared to state of the art.
Novelty: The novelty of the algorithm lies in providing global convergence guarantees for the online version of CCA. The algorithm has potential - however, this would have been better highlighted if real-applications were used (e.g. neuroscience experiments)  Impact/Significance: CCA is an important algorithm that finds applications in Cognitive Neuroscience, NLP, speech recognition, molecular biology, etc. To this end, having an efficient online algorithm for such settings is valuable.  Clarity: The authors do a great job presenting exactly the necessary information needed: from introducing CCA to the power method version of CCA, and to its ALS and stochastic formulation. At each step of the description, the authors provide remarks about what can go wrong: what happens when there is a gap, what about the normalization step (is it expensive?), what about input sparsity, etc. The flow of the paper is divided into small paragraphs/subsections, each of which provide an additional piece of information about the proposed algorithms. Really helpful for easy reading.  Other comments: 1. The authors should consider comparing also with the recent paper: Raman Arora, Poorya Mianjy and Teodor Marinov. Stochastic optimization for multiview learning using partial least squares. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2016. In their approach, the authors also consider a (partial) least-squares procedure for solution.  2. It would be helpful if the authors could describe in 1-2 sentences the findings, presented in Table 1. I.e., how much does log^2(1/eta) affect the performance over log(1/eta)? Is ASVRG always better than SVRG? If such overall comments are present, these will be really appreciated by readers. (Some of these comments are scattered over the text - maybe you could consider gathering all the comments together). 3. What is not clear from the text is whether the algorithm 1, which is taken from [7], wasn't proved to converge until now. If this is the first time this algorithm is proved, this should be stressed out in the text. If not, then the proof in the appendix is redundant and Theorem 1 should have a reference. This point is to discussed also with the reviewers, after the rebuttal. 4. How much significant is to set \epsilon(T) as proposed in Theorem 2? It seems that we need to know some information about the problem at hand (singular values, etc)?  5. It is not clear where the upper bound on the sub-optimality of initialization for SVRG is crucial for the proof of Lemma 3, or the overall analytical complexity of Algorithm 2. Can you please elaborate? 6. Maybe a small description of the technique shift-and-invert-preconditioning would be helpful to understand the idea behind it. Currently, the authors directly apply the steps, following similar notions from analogous PCA approaches.   Overall, this is a good paper and for the moment I have no reason to push for rejection. However, some re-writing of parts of the paper are needed for publication.