Large Scale Canonical Correlation Analysis with Iterative Least Squares
Canonical Correlation Analysis (CCA) is a widely used statistical tool with both well established theory and favorable performance for a wide range of machine learning problems. However, computing CCA for huge datasets can be very slow since it involves implementing QR decomposition or singular value decomposition of huge matrices. In this paper we introduce L-CCA , a iterative algorithm which can compute CCA fast on huge sparse datasets. Theory on both the asymptotic convergence and finite time accuracy of L-CCA are established. The experiments also show that L-CCA outperform other fast CCA approximation schemes on two real datasets.
I acknowledge having read the authors' feedback. Given the clarification on the novelty aspects, I still feel that the technical contributions are minor even though the eventual algorithm is both useful and interesting. The authors claim Section 2.3 and Algortihm 1 to be novel, but several alternating algorithms for solving CCA via linear regression have been presented before (I did not mention these in the original review because I assumed Section 2 to be considered as background). Besides [20] and [22], for example Dehon, Filzmoser and Croux ("Robust methods for canonical correlation analysis", 2000), and Lykou and Whittaker ("Sparse CCA using a Lasso with positivity constraints", 2010) have in recent years presented CCA algorithms based on such derivation, citing earlier works from the 60s and 70s to motivate the re-formulation. I did not check whether all of the details are identical, but at least the relationship between the proposed algorithm and these earlier alternating LS derivations should be discussed in detail, so that readers would be able to see the novelty. If the details are the same, 2.3 cannot be considered novel.I now understand that the LING part has novel elements as well, namely the support for sparse matrices. However, this is not at all apparent in the writing and the derivations are very similar to the UAI paper. While solving CCA is indeed an excellent use-case for that technique, it is very straightforward and LING could also be considered as background. The resulting method might well be the best choice for learning large-scale CCA models, but this would be better presented for the community by a paper that does not hide the proofs in the appendix (but instead illustrates somehow what they mean in practice) and that has much more extensive experimentation, such as explicitly showing how slow the naive algorithm is, comparison of running times on sparse and non-sparse data, better study of the conditions where RP-CCA works and fails, and illustration on how the performance of the proposed algorithm depends on the the parameters k_{pc}, t_1 and t_2 to provide practical suggestions.(The rest is the original review)	The paper proposes an algorithm for efficient learning of CCA	for large sparse matrices, by iteratively solving learst	squares problems.  Different alternatives proposed by the	authors are compared on two  data sets with fairly good	results.	I am puzzled by the role of LING in this publication.  It is	mentioned in the background section with a citation to a very	recent arXiv publication, yet it is explained in Section 3 as	part of the new algorithm. I would like the authors to clarify	the exact relationship between these papers.  This is a	critical question since the eventual algorith is a direct	application of LING to Algorithm 1, which is not new in	itself, and the error analysis of Theorem 3 alone is not quite	sufficient to warrant publication in a top forum, especially	as it has been pushed to the Supplement.	Following the recent large-scale NLP applications of CCA, it	makes sense to consider fast alternatives. The motivation	seems to be in particular for solving setups with large p, for	which the starndard solutions are said to be quadratic or	cubic. There are, however, also solutions that are either	constant wrt p (kernel CCA solved in the dual space, for	example Hardoon et al. Neuroimage'07 uses KCCA with linear	kernels that are equivalent to CCA and p is only relevant when	computing the kernel) or linear in p (the more recent	probabilistic solutions; for example Klami et al. ICLR'14	presents a variant for sparse matrices).  It would make sense	to discuss also the relationship with such algorithms, to	clarify why they would not be as fast in practice as the	proposed method (KCCA is slow for large N, whereas	probabilistic CCA should suffer from the same problem as	RP-CCA, requiring large k to capture correlations with small	variance).	I like the experiments in the sense that they attempt to	illustrate how different variants work in different	conditions.  Comparison against RP-CCA is also very relevant,	since I believe for example [7] uses something like that, and	your results clearly show that RP-CCA is very good in certain	conditions (Fig. 2(a)) and poor in others. This is a valuable	result.  However, I am somewhat disappointed with the way the	experiments are conducted.  First of all, merely computing	correlations on the training data provides a very limited view	to CCA, especially as CCA on high-dimensional data overfits	ridiculously easily. Even though you get higher correlations	on training data compared to RP-CCA, the results could be	opposite on test data since focusing on dimensions with high	variance could reduce overlearning. This would imply that	the singular vectors are also off.  Hence, I would strongly	recommend reporting also correlations computed on leave-out	test samples.	Also, the test setup where you report results for three fixed	CPU time choices does not convey a very clear message.  Why	not just plot the sums of the first 20 correlations as a	function of (logarithmic) time, by running each algorithm on	multiple choices of the parameters?  Then show the actual	values like in Fig 1 and 2 for one snapshot. At least for me	this would tell more about how the algoritghms work.  It would	also be nice to somehow illustrate just how slow the naive	algorithms would be, perhaps by extrapolating the	computational time from a smaller data.	Quality: The method seems technically correct and the experiments	are satisfactory, albeit not optimal for illustrating the algorithms	in practice.	Clarity: The paper is easy enough to read, and the algorithm	boxes are very useful.	Originality: The method is rather straightforward and has	limited originality, since it simply plugs in LING for solving	CCA.  In the end, we are merely talking about a gradient-based	algorithm for solving a linear method that is known to be	solvable with least squares algorithms.	Significance: The paper has practical signficance since there	is clear demand for CCA implementations applicable on very	large data. This would further be improved by good open-source	implementation of the algorithm, good rules of thumb for	choosing the parameters, and extensive experiments.  The	theoretical significance is, however, quite limited. Maybe	some other forum would be more suitable for this kind of a	practical contribution.