Learning Kernels with Random Features
Randomized features provide a computationally efficient way to approximate kernel machines in machine learning tasks. However, such methods require a user-defined kernel as input. We extend the randomized-feature approach to the task of learning a kernel (via its associated random features). Specifically, we present an efficient optimization problem that learns a kernel in a supervised manner. We prove the consistency of the estimated kernel as well as generalization bounds for the class of estimators induced by the optimized kernel, and we experimentally evaluate our technique on several datasets. Our approach is efficient and highly scalable, and we attain competitive results with a fraction of the training cost of other techniques.
This work presents a method to speed up kernel learning using random features (RF). With respect to known approaches, e.g. structured composition of kernels with respect to an alignment metric and joint kernel composition optimization based on empirical risk minimization, it achieves higher efficiency by optimizing compositions of kernels using explicit RF maps rather than full kernel matrices. The proposed method is divided in 2 steps: First, the kernel is learned efficiently using RF. Then, the optimized features associated to the learned kernel are used in a standard supervised learning setting to compute an estimator. Authors prove the consistency of the learned kernel and generalization guarantees for the learned estimator. An empirical evaluation of the method is also provided, including: - A toy example showing the learned features in the case of a bad initial kernel guess - An experiment involving a high-dimensional dataset, showing that the method attains higher predictive performance and induces a sparse representation, which can be potentially useful for interpretability - A performance evaluation on benchmark datasets, showing comparable test accuracies at a fraction of the computational cost with respect to the standard RF approach
The proposed approach is very interesting and novel. The idea of using random features to speed up kernel alignment is brilliant. The paper is well written and organized in a principled way. The authors provide a theoretical analysis of the method, guaranteeing consistency of the learned kernel and generalization properties of the resulting estimator. The experimental part allows to better visualize the method, shows how it can be used to identify sparse features in high dimensions and compares the accuracy and computational time of the proposed method on 3 benchmark datasets. Regarding the latter point, benchmarking experiments highlight promising practical applications for the method. As an improvement, I would suggest to evaluate the method on more benchmark datasets, for instance some of the ones used in [18].   Minor issues:   37 (?): optimization --> optimization problem   76, 83, 89: Properly introduce (or change) $W$ and $W^k$, highlighting the relationship with lower-case $w^i$  87: need only store --> only needs to store  98: missing ")" the last element of $z^i$  189: one-hot --> one-shot  Algorithm 1: Add "end" at the end of each while cycle  Figure 1: It is not clear why the optimized random features $w^k$ clustered around (-1,-1) and (1,1), depicted in yellow in the figure, would indicate a good solution.