Knockoff-Inspired Feature Selection via Generative Models
We propose a feature selection algorithm for supervised learning inspired by the recently introduced knockoff framework for variable selection in statistical regression. While variable selection in statistics aims to distinguish between true and false predictors, feature selection in machine learning aims to reduce the dimensionality of the data while preserving the performance of the learning method. The knockoff framework has attracted significant interest due to its strong control of false discoveries while preserving predictive power. In contrast to the original approach and later variants that assume a given probabilistic model for the variables, our proposed approach relies on data-driven generative models that learn mappings from data space to a parametric space that characterizes the probability distribution of the data. Our approach requires only the availability of mappings from data space to a distribution in parametric space and from parametric space to a distribution in data space; thus, it can be integrated with multiple popular generative models from machine learning. We provide example knockoff designs using a variational autoencoder and a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff designs and the knockoff framework with respect to existing approaches.
This paper presents an interesting use of a knockoff framework similar to that of model-X knockoffs (Candes et al., 2018) in generative models like VAE and GPLVM for feature selection in supervised learning. On the flip side, it is really hard to tell from the experimental results (Fig. 2) what performance benefits their proposed designs bring over the state of the art. I would encourage the authors to provide a more detailed analysis of the conditions under which their proposed designs would outperform the state of the art (or not). The author should consider presenting their results through other means that can better highlight the performance benefits of their proposed designs. Besides applying their knockoff designs to a softmax classifier, can the authors provide examples of applications to other supervised learning models? Proof of Lemma 1: Without further assumptions from that stated in the lemma, can the authors provide a derivation (or result that they*ve used) and explanation for the mean vector and covariance matrix of the joint probability p(z,x_n|x_{?n}) = p(z|x)p(x_n|x_{?n})? In particular, why are z and x_n conditionally independent given x_{-n}, as reflected in their zero covariance value? Page 2: The authors say that *they must be independent from the labels to be predicted*. Shouldn*t it be conditionally independent (2nd paragraph, page 3) instead? Minor issues Page 5: The notation of tilde{bold{x}}_n is ambiguous since the subscript is used to index the data sample (page 2). Is it intended to be without a bold? Step 3 of Algorithm 1: I would have preferred that it is stated consistently with that in the main text (step i in 2nd paragraph of Section 3). Page 5: This sentence is difficult to parse: *the training dataset for the generative models used to compute new knockoffs sample values is trained over the original data samples*. Page 5: SInce? Page 6: I cannot find Figure 1(g) and Figure 1(h).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Knockoff-Inspired Feature Selection via Generative Models
We propose a feature selection algorithm for supervised learning inspired by the recently introduced knockoff framework for variable selection in statistical regression. While variable selection in statistics aims to distinguish between true and false predictors, feature selection in machine learning aims to reduce the dimensionality of the data while preserving the performance of the learning method. The knockoff framework has attracted significant interest due to its strong control of false discoveries while preserving predictive power. In contrast to the original approach and later variants that assume a given probabilistic model for the variables, our proposed approach relies on data-driven generative models that learn mappings from data space to a parametric space that characterizes the probability distribution of the data. Our approach requires only the availability of mappings from data space to a distribution in parametric space and from parametric space to a distribution in data space; thus, it can be integrated with multiple popular generative models from machine learning. We provide example knockoff designs using a variational autoencoder and a Gaussian process latent variable model. We also propose a knockoff score metric for a softmax classifier that accounts for the contribution of each feature and its knockoff during supervised learning. Experimental results with multiple benchmark datasets for feature selection showcase the advantages of our knockoff designs and the knockoff framework with respect to existing approaches.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents an interesting use of a knockoff framework similar to that of model-X knockoffs (Candes et al., 2018) in generative models like VAE and GPLVM for feature selection in supervised learning. On the flip side, it is really hard to tell from the experimental results (Fig. 2) what performance benefits their proposed designs bring over the state of the art. I would encourage the authors to provide a more detailed analysis of the conditions under which their proposed designs would outperform the state of the art (or not). The author should consider presenting their results through other means that can better highlight the performance benefits of their proposed designs. Besides applying their knockoff designs to a softmax classifier, can the authors provide examples of applications to other supervised learning models? Proof of Lemma 1: Without further assumptions from that stated in the lemma, can the authors provide a derivation (or result that they*ve used) and explanation for the mean vector and covariance matrix of the joint probability p(z,x_n|x_{?n}) = p(z|x)p(x_n|x_{?n})? In particular, why are z and x_n conditionally independent given x_{-n}, as reflected in their zero covariance value? Page 2: The authors say that *they must be independent from the labels to be predicted*. Shouldn*t it be conditionally independent (2nd paragraph, page 3) instead? Minor issues Page 5: The notation of tilde{bold{x}}_n is ambiguous since the subscript is used to index the data sample (page 2). Is it intended to be without a bold? Step 3 of Algorithm 1: I would have preferred that it is stated consistently with that in the main text (step i in 2nd paragraph of Section 3). Page 5: This sentence is difficult to parse: *the training dataset for the generative models used to compute new knockoffs sample values is trained over the original data samples*. Page 5: SInce? Page 6: I cannot find Figure 1(g) and Figure 1(h).
assistant
This paper presents an interesting use of a knockoff framework similar to that of model-X knockoffs (Candes et al., 2018) in generative models like VAE and GPLVM for feature selection in supervised learning. On the flip side, it is really hard to tell from the experimental results (Fig. 2) what performance benefits their proposed designs bring over the state of the art. I would encourage the authors to provide a more detailed analysis of the conditions under which their proposed designs would outperform the state of the art (or not). The author should consider presenting their results through other means that can better highlight the performance benefits of their proposed designs. Besides applying their knockoff designs to a softmax classifier, can the authors provide examples of applications to other supervised learning models? Proof of Lemma 1: Without further assumptions from that stated in the lemma, can the authors provide a derivation (or result that they*ve used) and explanation for the mean vector and covariance matrix of the joint probability p(z,x_n|x_{?n}) = p(z|x)p(x_n|x_{?n})? In particular, why are z and x_n conditionally independent given x_{-n}, as reflected in their zero covariance value? Page 2: The authors say that *they must be independent from the labels to be predicted*. Shouldn*t it be conditionally independent (2nd paragraph, page 3) instead? Minor issues Page 5: The notation of tilde{bold{x}}_n is ambiguous since the subscript is used to index the data sample (page 2). Is it intended to be without a bold? Step 3 of Algorithm 1: I would have preferred that it is stated consistently with that in the main text (step i in 2nd paragraph of Section 3). Page 5: This sentence is difficult to parse: *the training dataset for the generative models used to compute new knockoffs sample values is trained over the original data samples*. Page 5: SInce? Page 6: I cannot find Figure 1(g) and Figure 1(h).