Column Selection via Adaptive Sampling
Selecting a good column (or row) subset of massive data matrices has found many applications in data analysis and machine learning. We propose a new adaptive sampling algorithm that can be used to improve any relative-error column selection algorithm. Our algorithm delivers a tighter theoretical bound on the approximation error which we also demonstrate empirically using two well known relative-error column subset selection algorithms. Our experimental results on synthetic and real-world data show that our algorithm outperforms non-adaptive sampling as well as prior adaptive sampling approaches.
The paper studies the column subset selection problem (CSSP) via adaptive sampling. The mains points are: 1) The adaptive sampling strategy developed in [8] achieves better reconstruction error than continued sampling for additive error CSSP-algorithms, but gives a worse reconstruction error guarantee than continued sampling for relative error CSSP-algorithms.2) The paper proposes a novel adaptive sampling strategy which is a subtle modification to the strategy presented in [8]. Instead of subtracting the full projection for computing the residual matrix, the proposed strategy in the l-th sampling round subtracts the best rank-lk approximation to the target matrix in the span of the columns selected from the first l-1 rounds. The theorem shows that the proposed strategy is able to achieve better theoretical guarantee over continued sampling for relative-error CSSP algorithms.3) The authors implemented the proposed strategy using two relative-error column selection algorithms. Experiments show that they outperform continued sampling and prior adaptive sampling approaches. Overall, I like this paper. It is well written and is a pleasure to read. Although the approach introduced in this paper is incremental to the existing literature, it is solid and technically sound. To me, this paper asks an interesting question and addresses it in a simple but elegant way. I think it should be of interest to a subset of the NIPS audience.On the negative side, the claim that data matrices have rapidly decaying singular values is not always true. For example, Bai-Yin's law points out that if the entries of a m*n (m >= n) matrix A are independent standard normal random variables, then almost surely the largest singular value is around sqrt(m)+sqrt(n), while the smallest singular value is around sqrt(m)-sqrt(n). This suggests that all the singular values of A are pretty similar if A is a tall-and-skinny matrix. Since the theoretical guarantee of this paper heavily depends on the rapid decay of the spectrum of the target matrix, I am afraid that the proposed approach will perform poorly when dealing with matrix with almost flat singular values. I would have liked to see some discussions about this issue.Minor issues: - Line 74: is t rounds -> in t rounds - Line 411: in[15] -> in [15] - In page 7, the font size in figures are not consistent.