Subspace Embeddings for the Polynomial Kernel
Sketching is a powerful dimensionality reduction tool for accelerating statistical learning algorithms. However, its applicability has been limited to a certain extent since the crucial ingredient, the so-called oblivious subspace embedding, can only be applied to data spaces with an explicit representation as the column span or row span of a matrix, while in many settings learning is done in a high-dimensional space implicitly defined by the data matrix via a kernel transformation. We propose the first fast oblivious subspace embeddings that are able to embed a space induced by a non-linear kernel without explicitly mapping the data to the highdimensional space. In particular, we propose an embedding for mappings induced by the polynomial kernel. Using the subspace embeddings, we obtain the fastest known algorithms for computing an implicit low rank approximation of the higher-dimension mapping of the data matrix, and for computing an approximate kernel PCA of the data, as well as doing approximate kernel principal component regression.
The paper provides an algorithm to find a subspace embedding for polynomial kernel.The authors claim to propose the first fast OSE for non-linear kernel. This statement appears confusing to me. First of all, there are works on approximating OSE for non-linear kernel, for example [1] below, which does not mention in the paper at all. And the algorithm in [1] could be used for Gaussian kernel. Thus I would suppose that this paper focuses on the efficiency of the algorithm. Regarding the main theorem, it does provide a clear explanation that tensor sketch, in terms of \phi(v) T could be understood as an OSE. I do not manage to read the whole proof, but the intuition seems to be right. One of the main issues I have in mind is about the 5. in the Algorithm 1. The authors mention that \phi(v) T could be computed efficiently by [3]. However, it is not so intuitive for me since the original paper is not about Tensor Sketch directly. I suggest the authors provides more details on this, at least in the supplementary materials, since this is an important part of the algorithms. Regarding the efficiency, it would be interesting it the authors provide some comparisons between other methods for OSE.In the experiment section, the authors make some contradicting statement with the claim about efficiency of the algorithm. For example, "K-space can be rather expensive both in time and in memory to compute". This really confuses me very much, since as I understand one important point about the algorithm is that it gives a subspace embedding in polynomial time. Moreover, the experiments shows that the features extracted by the algorithm improves the performance of supervised learning compared to using the original features. This result makes sense, but it is already well-known from previous works that OSE usually gives better performances than original features. One interesting experiment I want to see would be the comparison between the proposed algorithm and the existing OSE algorithms.Please also be more careful about the mathematical formula in the paper. For example, at line 302 I should be "V U \phi(R) - \phi(A)" if I understand correctly.[1]  Kernelized Locality-Sensitive Hashing