KNOCKOFFGAN: GENERATING KNOCKOFFS FOR FEATURE SELECTION USING GENERATIVE ADVER- SARIAL NETWORKS
Feature selection is a pervasive problem. The discovery of relevant features can be as important for performing a particular task (such as to avoid overfitting in prediction) as it can be for understanding the underlying processes governing the true label (such as discovering relevant genetic factors for a disease). Machine learning driven feature selection can enable discovery from large, high-dimensional, nonlinear observational datasets by creating a subset of features for experts to focus on. In order to use expert time most efficiently, we need a principled methodology capable of controlling the False Discovery Rate. In this work, we build on the promising Knockoff framework by developing a flexible knockoff generation model. We adapt the Generative Adversarial Networks framework to allow us to generate knockoffs with no assumptions on the feature distribution. Our model consists of 4 networks, a generator, a discriminator, a stability network and a power network. We demonstrate the capability of our model to perform feature selection, showing that it performs as well as the originally proposed knockoff generation model in the Gaussian setting and that it outperforms the original model in non-Gaussian settings, including on a real-world dataset.
This manuscript describes an extension of the knockoff framework, which is designed to carry out feature selection while controlling the FDR among selected features, to settings in which the generative distribution of the features is not Gaussian. Specifically, the authors employ a GAN (with several modifications and additions) in which the generator produces knockoffs and the discriminator attempts to identify which features have been swapped between the original and the knockoff.
The method works as follows: (1) A conditional generator takes random noise and the real features as input, and outputs knockoff features. (2) A modified discriminator is used in such a way that the generator learns to generate knockoffs satisfying the necessary swap condition, so as to control the FDR of the knockoff procedure. (3) A power network uses Mutual Information Neural Estimation (MINE) to estimate the mutual information between each feature and its knockoff counterpart, so as to maximize the power of the knockoff procedure.
Results are provided on synthetic and real data. As for the synthetic data, when the underlying feature distribution is Gaussian, the proposed method, KnockoffGAN, performs almost as well as the original knockoff and outperforms the BHq method; when the underlying feature distribution is non-Gaussian, KnockoffGAN dominates both the original knockoff and BHq methods. As for the real data, the authors claim to identify nine relevant features for cardiovascular disease and eight relevant features for diabetes, whereas the original knockoff procedure identifies zero features from the same data.
General comments: 
This is an extremely impressive piece of work. The manuscript itself is a pleasure to read, and the results clearly demonstrate that the proposed KnockoffGAN both controls FDR and achieves power comparable to the original knockoff procedure in the Gaussian setting and much better than the original knockoff when the underlying distribution is not Gaussian.
Strengths: 
The combination of GANs and knockoff filter is a very promising and intriguing idea.
The use of the modified discriminator to ensure that the generated knockoffs satisfy the necessary swap condition is novel and intuitively sound.
The use of MINE to maximize power by maximizing the mutual information between each feature and its knockoff counterpart is also interesting.
The paper is well written, reads smoothly and the ideas are well exposed.
The illustrative figure is straightforward.
Weaknesses:
Intuitively, the modified discriminator and the power network should conflict with each other. I expect it was tricky to achieve a good tradeoff between two, but the authors failed to elaborate on these details.
The authors do not provide the design details of the neural networks. How dependent on the specific parametrization of the network architecture is the performance? How does the training order of four networks matter to the performance? 
The manuscript should cite [[ Jaime Roquero Gimenez, Amirata Ghorbani, and James Zou. "Knockoffs   for the mass: new feature importance statistics with false discovery   guarantees." arXiv:1807.06214, 2018. ]] which proposes a way to generate knockoffs for a Gaussian mixture model, and this method should be included in the relevant supplementary figure.
In Section 5.1.4, I would like to know, for a fixed data set, how the regularization affects the final values of the other loss terms.
The analysis of real data in Section 5.2 is unsatisfying in several respects.
First, there is an unfortunate oversight in Table 1: the text refers to three features that are "trivial," but only one of these is marked with an asterisk.  This leaves open the question of whether there are other trivial features beyond the three mentioned in the text. In addition, it is not clear exactly what it means for a feature to be "trivial" in this context.
This point gets to a deeper problem with the evaluation, which is that we are told, with no evidence, that these features are supported by literature in PubMed.  I would like to see two things here.  First, it seems obvious to me that if you are going to say that there is support in PubMed, you are obliged to actually report the citations that supposedly give this support. This could be done in the appendix. Equally importantly, there is a potential here for ascertainment bias which should be combatted in some fashion.  Presumably, some human expert had to do the PubMed searches to make this assessment.  I would like to know how "permissive" this assessor is.  To assess this, one could give the assessor a collection of terms, some of which were selected by KnockoffGAN and some at random, and then report the results. Obviously, some features that are significant may not be in the list of selected terms (because KnockoffGAN does not achieve 100% power) and so may appear as false negatives. But without some assessment like this, I have trouble believing this assessment.
A related point is that it seems quite unfortunate that the authors chose a data set that cannot be described at all due to the anonymity constraint.  At the very least, it seems that we should be told the dimensionality of the data set. The Knockoff literature contains real data sets that could have been used here.
Minor comments:
On the first page, the sentence beginning "On the other hand," should clarify that this is only in expectation.
p. 3: Missing right paren after [7].
p. 5: Write out “Gaussian process.”
p. 5: "as little" -> "as little as possible"
p. 6: "to show that in" -> "to show, in"
In Figures 2-5, add a horizontal line at 10% FDR for reference.
p. 10: "features ones" -> "features"
Note to program committee:
I did not review the technical details of the proof in the appendix.