Nonparametric von Mises Estimators for Entropies, Divergences and Mutual Informations
We propose and analyse estimators for statistical functionals of one or more distributions under nonparametric assumptions. Our estimators are derived from the von Mises expansion and are based on the theory of influence functions, which appear in the semiparametric statistics literature. We show that estimators based either on data-splitting or a leave-one-out technique enjoy fast rates of convergence and other favorable theoretical properties. We apply this framework to derive estimators for several popular information theoretic quantities, and via empirical evaluation, show the advantage of this approach over existing estimators.
Summary of paper: This paper presents a leave-one-out (LOO) estimator for a functional of a single distribution and second LOO estimator for a functional of two distributions. The authors present analysis for these LOO estimators as well as for data split (DS) estimators. Lastly, the paper presents experiments on synthetic datasets and an benchmark dataset. The authors have many goals for this paper from introducing new ideas to the machine learning community to proofs and experiments. It feels as though there could be two papers here: one for functionals of a single distribution and a second for functionals of two distributions. By splitting this into two papers, the authors would have more space in the main paper to add additional exposition detailing the purpose of each idea introduced, instead of relying so heavily on the supplemental material. This paper is a challenging read. The authors have several moments of clarification that are a welcome break from the deluge of technical functional analytic details. For example, the last sentence of the first paragraph does an excellent job summarizing the majority of the paper. Further examples of helpful clarification are the discussions following the list of contributions (starting at line 75) and Theorem 8 (starting at line 308). In contrast, at the top of the second page, the authors state that they will "introduce the calculus of influence functions to the machine learning community." While the authors do provide a technical definition for influence functions, they do not provide approachable exposition on influence functions, explaining what they are, what they can be used for, or why one may use them. Despite reading the paper several times, I had to seek out additional resources to gain a reasonable understanding of influence functions. There are several missed opportunities for short exposition about influence functions, such as at line 85 or following Definition 2 at line 111. The writing style and organization of the paper conceal the machine learning advancements and how these advances differ from the previous work. For example, by placing the "Comparison with Other Approaches" at the near end of the paper, a reader may be mislead into thinking that the work presented in sections 2 through 4 are either the work of the authors or standard in this sub-field of influence functions. After reading section 5, I was aware of the differences between this work and previous work and thus, I was better prepared to re-read sections 2 through 4 than I was after reading those sections the first time. I recommend both moving section 5 to earlier in the paper (either as part of section 1 or immediately following section 1) and to add a bit of exposition at the beginning of section 3 "Estimating Functionals" that makes it clear where the authors contributions begin. There are many impressive facets of this work. The results on the ETH-80 dataset are impressive in context of the previous work presented in this paper. I wish there had been additional exposition about this experiment in the main paper. I also like that they present an open question (at line 267) that is a result of their work. This makes it clear that their work has pushed the boundary for the field and has opened further research avenues.