We propose a feature selection algorithm for supervised learning inspired by the recently introduced knockoff framework for variable selection in statistical regression. While variable selection in statistics aims to distinguish between true and false predictors, feature selection in machine learning aims to reduce the dimensionality of the data while preserving the performance of the learning method. The knockoff framework has attracted significant interest due to its strong control of false discoveries while preserving predictive power. In contrast to the original approach and later variants that assume a given probabilistic model for the variables, our proposed approach relies on data-driven generative models that learn mappings from data space to a parametric space that characterizes the probability distribution of the data. Our approach requires only the availability of mappings from data space to a distribution in parametric space and from parametric space to a distribution in data space; thus, it can be integrated with multiple popular generative models from machine learning. We provide example knockoff designs using a variational autoencoder and a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff designs and the knockoff framework with respect to existing approaches.
This paper proposes a way of employing modern generative models like VAEs and GPLVMs within the recently popular “knockoff” framework for variable/feature selection. Roughly speaking, the main idea of knockoffs is to construct a duplicate (“knockoff”) of each regression variable which matches its distributional properties but is independent of the response variable when conditioning on the true input variable the knockoff is based on. Under certain assumptions, fitting a model on top of both original and knocked-off features and observing the difference between the fitted coefficients for the true and the knockoff features can then be used for variable selection with guaranteed false discovery rate. The authors of this paper suggest that many of the current methods for construction of knockoffs may not be appropriate for image, text, and other types of datasets commonly used in modern ML, and suggest a heuristic way of employing VAEs and GPLVMs for this purpose. The paper concludes with an empirical study which shows that their algorithm is competitive with existing feature selections methods in terms of post-selection accuracy of the fitted model.
I am currently leaning towards recommending rejection. While the paper is nicely written and does a good job of reviewing knockoffs, I see two main issues: (1) I am not sure about applications for the proposed algorithm; in particular, the authors allude to use on devices with limited memory and computational power, but do not discuss why the Johnson-Lindenstrauss transform or some of the many low-precision implementations of neural networks (binarised neural networks, xnor-nets, …) cannot be used; furthermore, in the case of images, I am not sure why there is no comparison to simple downsampling to a smaller resolution; (2) The reported results do not show a reliable improvements over existing methods.
Major comments:
- I am not entirely sure lemma 1 is correct. In particular, mu_{z | x} tends to be a function of the whole vector x including x_n. For example, if (a, b) are jointly distributed according to a bivariate normal, then E (b | a) = E(b) + Sigma_{ab} Sigma_{aa}^{-1} (a - E(a)) where Sigma is the corresponding covariance matrix. Hence claiming that the marginal distribution z | x_{-n} is N ( mu_{z|x} , Sigma_{z | x} ) seems wrong as mu_{z | x} will generally depend on x_n (similarly to how E(b | a) depends on “a” above) which cannot be the case when x_n is marginalised. If z depends linearly on x, there is a standard expression for the distribution you seek. However, with the non-linear dependence employed, e.g., within VAE, working out a closed form expression may be quite a challenge. Can you please clarify or drop this result from your paper if it indeed turns out incorrect?
- Can you please clarify why don’t you benchmark against the cited algorithm proposed in Lu et al. (2018)?
- Can you please explain how was the latent dimensionality for the VAEs (5) and GPLVMs (10) selected? Also, for the algorithms where random seed plays a role (e.g., VAEs), how many random seeds were used (are the numbers reported in fig.2 a result of averaging over multiple seeds)? Relatedly, have you tested whether starting from different seeds results in the same subset of variables selected (e.g., when using VAEs)?
Minor comments:
- In the 1st sentence of the introduction, “prevalence” can be high or low, increase or decrease, etc. but “becoming increasingly pervasive” does not sound right. Please consider rewording.
- In fig.1e, do you know why GPLVM leads to such a significant mode collapse?
- Just after the 1st display on p.5, “SInce” -> “Since”