POST SELECTION INFERENCE WITH INCOMPLETE MAXIMUM MEAN DISCREPANCY ESTIMATOR
Measuring divergence between two distributions is essential in machine learning and statistics and has various applications including binary classification, change point detection, and two-sample test. Furthermore, in the era of big data, designing divergence measure that is interpretable and can handle high-dimensional and complex data becomes extremely important. In this paper, we propose a post selection inference (PSI) framework for divergence measure, which can select a set of statistically significant features that discriminate two distributions. Specifically, we employ an additive variant of maximum mean discrepancy (MMD) for features and introduce a general hypothesis test for PSI. A novel MMD estimator using the incomplete U-statistics, which has an asymptotically normal distribution (under mild assumptions) and gives high detection power in PSI, is also proposed and analyzed theoretically. Through synthetic and real-world feature selection experiments, we show that the proposed framework can successfully detect statistically significant features. Last, we propose a sample selection framework for analyzing different members in the Generative Adversarial Networks (GANs) family.
The authors focus on the selection problem of k statistically significant features discriminating 2 probability distributions accessible via samples. They propose a non-parametric approach under the PSI (post selection inference) umbrella using MMD (maximum mean discrepancy) as a discrepancy measure between probability distributions. The idea is to apply (asymptotically) normal MMD estimators, rephrase the top-k selection problem as a linear constraint, and reduce the problem to Lee et al., 2016. The efficiency of the approach is illustrated on toy examples and in GAN (generative adversarial network) context. The technique complements the PSI-based independence testing approach recently proposed by Yamada et al., 2018. 
The submission is a well-organized, clearly written, nice contribution; it can be relevant to the machine learning community.
Below I enlist a few suggestions to improve the manuscript:
-Section 1: The notion of characteristic kernel (kernel when MMD is metric) has not been defined, but it was referred to. 'Due to the mean embeddings in RKHS, all moment information is stored.': This sentence is somewhat vague.
-Section 1: 'MMD can be computed in closed form'. This is rarely the case (except for e.g. Gaussian distributions with Gaussian or polynomial kernels). I assume that the authors wanted refer to the estimation of MMD.
-Section 1: 'K nearest neighbor approaches (Poczos & Schneider, 2011)'. The citation to this specific estimator can go under alpha-divergences. The Wasserstein metric could also be mentioned.
-Section 3.1: k is used to denote the number of selected features and also the kernel used in MMD. I suggest using different notations.
-Theorem 1: '\Phi is the CDF...'. There is no \Phi in the theorem.
-Section 3.2: The existence of MMD (mean embedding) requires certain assumptions: E_{x\sim p}\sqrt{k(x,x)} < \infty, E_{x\sim q}\sqrt{k(x,x)} < \infty.
-Section 3.2.: block estimator: 'B_1 and B_2 are finite'. 'fixed'?
-Section 3.2.: MMD_{inc}: 
   i) 'S_{n,k}': k looks superfluous.
   ii) 'l': it has not been introduced (cardinality of D).
-Section 3.3: typo: 'covraiance' (2x)
-Section 3.3: Fan et al. 2013: The citation can go to \citep{}.  
-Theorem 2: 
   i)'c' is left undefined.
   ii)Comma is missing before 'where'.
   iii)\xrightarrow{d} (Theorem 2, Corollary 3-4): Given that 'd' also denotes dimension in the submission, I suggest using a different notation for convergence in distribution.
-At the introduction of block-MMD the block size (B) was fixed, while in the experiments (e.g. Figure 3) it is growing with the sample size (B=\sqrt{n}). The assumption on B should be clearly stated.
-Section 5.1: (b) mean shift: comma is missing before 'where'.
-References: 
   i) Abbreviations and names in the titles should be capitalized (such as cramer, wasserstein, hilbert-schmidt, gan, nash). 
   ii) Scholkopf should be Sch\{"o}lkopf (in the ALT 2005 work).
   iii) 'Exact post-selection inference, with application to the lasso': All the authors are listed; 'et al.' is not needed.