Statistical Model Criticism using Kernel Two Sample Tests
We propose an exploratory approach to statistical model criticism using maximum mean discrepancy (MMD) two sample tests. Typical approaches to model criticism require a practitioner to select a statistic by which to measure discrepancies between data and a statistical model. MMD two sample tests are instead constructed as an analytic maximisation over a large space of possible statistics and therefore automatically select the statistic which most shows any discrepancy. We demonstrate on synthetic data that the selected statistic, called the witness function, can be used to identify where a statistical model most misrepresents the data it was trained on. We then apply the procedure to real data where the models being assessed are restricted Boltzmann machines, deep belief networks and Gaussian process regression and demonstrate the ways in which these models fail to capture the properties of the data they are trained on.
The function that attains the supremum of the statistic used in MMD two-sample tests can be used to identify regions in attribute space in which a statistical model most misrepresents the data used for automatic induction. This "witness function" can be interpreted as the difference of two kernel density estimates one corresponding to the actual data and one to the data generated with the fitted model. The authors then apply the procedure to real and synthetic learning problems and illustrate (rather than demonste, as the authors claim) the ways in which automatically induced models, such as restricted Boltzmann machines, deep belief networks or Gaussian processes regressors fail to capture the properties of the data on which they are trained.The paper is very well written and is a pleasure to read. However the originality and significance of the contribution is rather limited. Part of the problem stems from the lack of a clear focus: it is presented as a "an exploratory approach" that touches on many aspects of Statistical Model Criticism but fails to provide sufficient background or depth in the discussion of the issues addressed. This would be resolved in a longer version that provides a more detailed discussion.  According to the authors, the main contribution of this work is highlighting the importance for model criticism of the (known) property that the witness function in MMD two-sample tests is the difference of two kernel density estimates. This releveance apparently stems from the fact that the MMD statistic is the supremum among a large space o possible statistics. Leaving aside the connection with MMD, the witness function is simply the difference of kernel density estimates. If one were asked to identify regions in which a statistical model most misrepresents the data it was trained on, this method is probably the most obvious choice. In criticising a model one should have the use of the model in mind. Therefore, model evaluation ought to be guided by a utility function. I agree with the authors that focusing on the differences of kernel density estimates for the actual data and for data generated by the fitted model is probably an excellent general purpose model criticism tool, especially if I am interested in local differences. I would argue (rather trivially) that the important feature is the local character of RBF kernels. The maximization property is reassuring, albeit not of much practical use:The crucial issue seems to be the choice of kernel and kernel parameters. Unfortunately, in MMD two-sample tests, these need to be specified by the user.In summary, the paper is based on a fairly straightforward result: that differences of kernel densitie estimates help us understand how and where a model fails to fit the data. The experimental investigation is promising but seems to be rather preliminary and unsystematic. On the question posed by the authors: would it be worth this paper being published now as a conference paper to bring model criticism to the attention of the machine learning community quickly or waiting a year or two for a review article?Yes, it would be worth waiting.   A minor comment:"While it is clear that simple methods will fail to capture all of the structure in this data, it is not clear a priori how much better the more advanced methods will fair."In line 297, the authors probably meant "will fare".