Informative Features for Model Comparison
Given two candidate models, and a set of target observations, we address the problem of measuring the relative goodness of fit of the two models. We propose two new statistical tests which are nonparametric, computationally efficient (runtime complexity is linear in the sample size), and interpretable. As a unique advantage, our tests can produce a set of examples (informative features) indicating the regions in the data domain where one model fits significantly better than the other. In a real-world problem of comparing GAN models, the test power of our new test matches that of the state-of-the-art test of relative goodness of fit, while being one order of magnitude faster.
I have read the author response, and in particular, the authors have clarified my confusion regarding Figure 2. While I maintain my overall evaluation, I do feel that the contributions of this paper would be strengthened if either more theoretical results are provided for the proposed tests, or if the experiment results could be more convincing (as noted by another reviewer).  ----------  The paper proposes two nonparametric statistical tests of relative goodness-of-fit that run in near-linear time, based on the unnormalized mean embeddings (UME) statistic and the finite-set Stein discrepancy (FSSD). By optimizing test locations to maximize test power, the proposed tests also yield informative features for distinguishing regions of the data domain in which one model fits significantly better than another. Experiments are conducted on toy problems and in comparing GAN models.  Overall, the paper is fairly clearly written and easy to follow. While nonparametric goodness-of-fit testing has received much attention recently, few works investigate the topic of relative goodness-of-fit testing, which could be important for model comparison in practice, since “all models are wrong, but some are useful.” In this sense, I feel that this work is a nice addition to the existing literature.   On the other hand, in terms of novelty, most of the techniques utilized in this work were developed in existing works: the definition of the UME and FSSD statistics as well as the idea of optimizing test locations to maximize test power in order to obtain informative features and reduce the computational complexity to near-linear time were all pioneered by Chwialkowski et al. and Jitkrittum et al., and the procedure to adjust the test threshold is similar to that in Bounliphone et al. The proofs of Theorems 1 and 2 are also fairly standard. I feel that the contributions of this paper would be strengthened if more theoretical results could be provided for the proposed Rel-UME, REl-FSSD tests in addition to deriving the asymptotic null distribution, such as an analysis of (asymptotic) test power which has been done for both the original UME and FSSD tests as well as MMD.  Regarding the experiments, they seem fairly extensive and well-executed, although the presentation of the results (in particular the figures and table) could be improved. In particular, I found the “Rejection rate” label on the y-axes in the figures confusing at times, and it would be clearer to explicitly specify “Type I” or “Type II” error depending on the circumstance. Figure 4(a) is not vey informative, and it would be more interesting to zoom into the smaller sample-size region. Figure 2 (b) and (c) seem to be have been flipped (at least according to the description in the text); I also don’t get why Figure 2(b) is dominated by cat images with only a few airplane images---shouldn’t it consist mostly of airplane images since these are which the distribution does not fit well? The results presented in Table 1 also indicates that the current GAN comparison task might be too easy and it is hard to distinguish the performance difference of the various methods; perhaps it would be more informative to mimic Experiment 1 and consider setting the distribution R to be a mix of S and N images with varying proportions as well.