Sublinear Time Low-Rank Approximation of Distance Matrices
Let P = {p 1 , p 2 , . . . p n } and Q = {q 1 , q 2 . . . q m } be two point sets in an arbitrary metric space. Let A represent the m × n pairwise distance matrix with A i,j = d(p i , q j ). Such distance matrices are commonly computed in software packages and have applications to learning image manifolds, handwriting recognition, and multi-dimensional unfolding, among other things. In an attempt to reduce their description size, we study low rank approximation of such matrices. Our main result is to show that for any underlying distance metric d, it is possible to achieve an additive error low rank approximation in sublinear time. We note that it is provably impossible to achieve such a guarantee in sublinear time for arbitrary matrices A, and our proof exploits special properties of distance matrices. We develop a recursive algorithm based on additive projection-cost preserving sampling. We then show that in general, relative error approximation in sublinear time is impossible for distance matrices, even if one allows for bicriteria solutions. Additionally, we show that if P = Q and d is the squared Euclidean distance, which is not a metric but rather the square of a metric, then a relative error bicriteria solution can be found in sublinear time. Finally, we empirically compare our algorithm with the singular value decomposition (SVD) and input sparsity time algorithms. Our algorithm is several hundred times faster than the SVD, and about 8-20 times faster than input sparsity methods on real-world and and synthetic datasets of size 10 8 . Accuracy-wise, our algorithm is only slightly worse than that of the SVD (optimal) and input-sparsity time algorithms.
and Demaine et al. use them for music and musical rhythms [7]. For an excellent tutorial on Euclidean distance matrices, we refer the reader to [8], which lists applications to nuclear magnetic resonance (NMR), crystallagraphy, visualizing protein structure, and multi-dimensional unfolding.
We consider the most general case for which P and Q are not necessarily the same set of points. For example, one may have two large unordered sets of samples from some distribution, and may want to determine how similar (or dissimilar) the sample sets are to each other. Such problems arise in hierarchical clustering and phylogenetic analysis 1 . Formally, Let P = {p 1 , p 2 , . . . p m } and Q = {q 1 , q 2 . . . q n } be two sets of points in an arbitrary metric space. Let A represent the m × n pairwise distance matrix with A i,j = d(p i , q j ). Since the matrix A may be very large, it is often desirable to reduce the number of parameters needed to describe it. Two standard methods of doing this are via sparsity and low-rank approximation. In the distance matrix setting, if one first filters P and Q to contain only distinct points, then each row and column can contain at most a single zero entry, so typically such matrices A are dense. Low-rank approximation, on the other hand, can be highly beneficial since if the point sets can be clustered into a small number of clusters, then each cluster can be used to define an approximately rank-1 component, and so A is an approximately low rank matrix.
To find a low rank factorization of A, one can compute its singular value decomposition (SVD), though in practice this takes min(mn 2 , m 2 n) time. One can do slightly better with theoretical algorithms for fast matrix multiplication, though not only are they impractical, but there exist much faster randomized approximation algorithms. Indeed, one can use Fast Johnson Lindenstrauss transforms (FJLT) [16], or CountSketch matrices [4,13,15,1,5], which for dense matrices A, run in O(mn) + (m + n)poly(k/ ) time. At first glance the O(mn) time seems like it could be optimal. Indeed, for arbitrary m × n matrices A, outputting a rank-k matrix B for which
can be shown to require Ω(mn) time. Here A k denotes the best rank-k approximation to A in Frobenius norm, and recall for an m × n matrix C, C 2 F = i=1,...,m,j=1,...,n C 2 i,j . The additive error guarantee above is common in low-rank approximation literature and appears in [10]. To see this lower bound, note that if one does not read nearly all the entries of A, then with good probability one may miss an entry of A which is arbitrarily large, and therefore cannot achieve (1.1).
Perhaps surprisingly, [14] show that for positive semidefinite (PSD) n × n matrices A, one can achieve (1.1) in sublinear time, namely, in n · k · poly(1/ ) time. Moreover, they achieve the stronger notion of relative error, that is, they output a rank-k matrix B for which
(1.
2)
The intuition behind their result is that the "large entries" causing the Ω(mn) lower bound cannot hide in a PSD matrix, since they necessarily create large diagonal entries. A natural question is whether it is possible to obtain low-rank approximation algorithms for distance matrices in sublinear time as well. A driving intuition that it may be possible is that no matter which metric the underlying points reside in, they necessarily satisfy the triangle inequality. Therefore, if
is large, and further, all these distances can be found by reading the first row and column of A. Thus, large entries cannot hide in the matrix. Are there sublinear time algorithms achieving (1.1)? Are there sublinear time algorithms achieving (1.2)? These are the questions we put forth and study in this paper.
This paper presents an algorithm for computing a near-optimal low-rank approximation for a distance matrix — i.e. a matrix where the (i,j)th entry equals d(p_i, q_j) for some metric d and sets of points p_1, …, p_m and q_1,… q_n. Distance matrices are common in learning and data analysis tasks and low-rank approximation is a ubiquitous approach to compressing matrices.   In general, the fastest existing low-rank approximation algorithms are randomized and, to compute a k-rank approximation, run in O(nnz(A) + min(n,m)*poly(k/eps)) time for a matrix A with nnz(A) non-zeros. Since the second term is often considered lower order (k is typically << n and eps is moderate), these algorithms run in roughly linear time in the size of the input.   Naturally, this is optimal for general matrices — if I don’t at least read every entry in the matrix, I could miss a very heavy entry that I would need to see to output a good low-rank approximation. However, we could ask if better runtimes are possible for matrices with more structure, like distance matrices. We might hope for algorithms that run in sublinear time. Recent work shows that, surprisingly, sublinear time low-rank approximation algorithms can be obtained for positive semidefinite matrices, another structured class of matrices. This result relies on the fact that it’s not possible to add an arbitrarily heavy element A_i,j to a PSD matrix without increasing the size of either A_i,i or A_j,j. Accordingly, simply reading the diagonal of A (which only takes n time) avoids the trivial lower bounds that apply to general matrices.   The authors begin with the observation that a similar statement is true for distance matrices. By triangle inequality, if A_i,j is very heavy, A_1,i, A_1,j, or A_1,1 must be too. These entries can all be found by reading the first row of A (or really any single row of A_. As in the case of PSD matrices, this property allows us to avoid lower bounds for general matrices.  The authors leverage this observation to motivate their algorithm, which is based on solving the low-rank approximation using a random subsample of the original matrix. In particular, they sample rows/columns by their norms. It is well known that subsampling in this way makes it possible to obtain a low-rank approximation within additive error eps*||A||_F^2 of optimal. For general matrices, computing norms naively takes nnz(A) time. However, the authors show that for distance matrices, norms can be estimated coarsely in sublinear time. They can subsample the matrix using these coarse estimates, and then recurse to shrink the problem size even smaller. In recursing, the authors alternate between row and column sampling.  Overall, this yields an algorithm that runs in (min(N,m)^1+gamma) * poly(k/eps)^1/gamma time for any chosen gamma, which controls the levels of recursion in the algorithm. The authors present a simpler algorithm with just one level of recursion that runs in min(n,m)^1.34 * poly(k/eps) time. The authors implement this algorithm and it turns out to be quite practical, outperforming nnz(A) time random projection methods pretty significantly.   In addition to their main result, the authors include a number of additional results. Of particular interest is a lower bound, which demonstrates that additive ||A||_F^2 error is necessary — achieving full relative error for the low-rank approximation problem isn’t possible even for distance matrices without at least reading the whole matrix. This places the complexity of low-rank approximation for distances matrices somewhere between standard matrices, where even additive error is not possible in sublinear time, and PSD matrices, where full relative error is possible in sublinear time.  Overall, I think the paper is well written and the techniques are new and interesting. It’s also nice to see that the main algorithm presented is already practical.   One concern with the paper is that I find the proof of the faster (min(N,m)^1+gamma) * poly(k/eps)^1/gamma difficult to follow. In particular, recursive sampling cannot be performed naively because, after sampling and reweighting rows from a distance matrix, you end up with a matrix that is no longer a distance matrix. To deal with this challenge, the authors suggest partitioning sampled rows into “buckets” based on how much they were reweighed by. The matrix to be recursed on is a valid distance matrix, at least when restricted to rows in one bucket, which allows column norms to be estimated by estimating the norm in each block of rows separately (estimating column norms is need for the next step of reduction).   This makes perfect sense to me for the two step in(n,m)^1.34 * poly(k/eps) time algorithm (Algorithm 3), but I do not understand how the approach works for the faster Algorithm 2. In particular, I don’t see how the argument carries through multiple levels of recursion without blowing up the number of buckets and, for whatever reason, Algorithm 2 does not include the reweighing step in its pseudocode! I would encourage the authors to add it in explicitly. If this makes the algorithm much more difficult to present, I would encourage the authors to present the simpler, but still interesting Algorithm 3 in the main part of their paper, and to give a full and rigorous development of Algorithm 2 in the appendix.