Minimax Estimation of Neural Net Distance
An important class of distance metrics proposed for training generative adversarial networks (GANs) is the integral probability metric (IPM), in which the neural net distance captures the practical GAN training via two neural networks. This paper investigates the minimax estimation problem of the neural net distance based on samples drawn from the distributions. We develop the first known minimax lower bound on the estimation error of the neural net distance, and an upper bound tighter than an existing bound on the estimator error for the empirical neural net distance. Our lower and upper bounds match not only in the order of the sample size but also in terms of the norm of the parameter matrices of neural networks, which justifies the empirical neural net distance as a good approximation of the true neural net distance for training GANs in practice.
This paper considers the minimax estimation problem of neural net distance. The problem originates from the generative adversarial networks (GANs). In particular, the paper established the lower bound on the minimax estimation for neural net distance which seems to the first result of this kind.  The authors then derived an upper bound on the estimation error which matches the minimax lower bound in terms of the order of sample size, and the norm of the parameter matrices. However, there is a gap of $\sqrt{d}$ or $\sqrt{d+h}$ which remains as an open question. The main techniques root from a critical lemma called Le Cam's methods in nonparametric statistics and the recent paper [26].    The upper bound used the Rademacher complexity and the extension of McDiarmid's inequality for unbounded support distributions and sub-gaussian.   Overall, the results are novel and very interesting in the setting of neural net distance. Several comments below:   1. The form of neural net distance is very similar to the MMD work in [26] where the same techniques (Le Cam's method) were used for deriving the minimax lower bound. It would be nice to explain the difference between this study and the techniques used in [26].    2. There is a gap between the minimax lower bound and the upper bound used. Any comments on this?  Is this because  the hypothesis space of deep network is too large for the neural net distance?     3.  The notation $\mathal{F}_{nn}$ for representing the function class of neural network could be confusing since here  $n$ denotes the number of samples    I am quite satisfied with the authors' feedback.  It is a good paper.