Near-Optimal Density Estimation in Near-Linear Time Using Variable-Width Histograms
Let p be an unknown and arbitrary probability distribution over [0, 1). We consider the problem of density estimation, in which a learning algorithm is given i.i.d. draws from p and must (with high probability) output a hypothesis distribution that is close to p. The main contribution of this paper is a highly efficient density estimation algorithm for learning using a variable-width histogram, i.e., a hypothesis distribution with a piecewise constant probability density function. In more detail, for any k and ", we give an algorithm that makesÕ(k/" 2 ) draws from p, runs inÕ(k/" 2 ) time, and outputs a hypothesis distribution h that is piecewise constant with O(k log 2
(1/")) pieces. With high probability the hypothesis
, where d TV denotes the total variation distance (statistical distance), C is a universal constant, and opt k (p) is the smallest total variation distance between p and any k-piecewise constant distribution. The sample size and running time of our algorithm are optimal up to logarithmic factors. The "approximation factor" C in our result is inherent in the problem, as we prove that no algorithm with sample size bounded in terms of k and " can achieve C < 2 regardless of what kind of hypothesis distribution it uses.
Near-optimal density estimation in near-linear time using variable-width histogramsGiven a univariate distribution with support on the unit interval, this paper develops a method for estimating the density of the distribution via a histogram. The histogram has the following features: the bins are data-dependent, the bins are determined efficiently (near linear time), the number of bins k is specified by the user, and the estimator satisfies a "probably approximately correct" type performance guarantee. The paper is entirely theoretical. Pros: The writing is clear. I believe the results to be novel and theoretically pretty interesting. Cons:  Some experiments would not have hurt the paper. A drawback of the theoretical framework is that model order k is not adaptively chosen.My main quibble is with the omission of some relevant references. In particular, the algorithm developed by this paper has a course-to-fine kind of strategy, and harkens to multiscale and wavelet-based methods for density estimation. Here are two relevant references:Rebecca Willett and Robert Nowak, "Multiscale Poisson Intensity and Density Estimation" IEEE Transactions on Information Theory, vol. 53, no. 9, pp. 3171-3187, 2007.J. Klemelä. (2009). Multivariate histograms with data-dependent partitions. Statistica Sinica. 19(1): 159-176. I would recommend the authors do both forward and reverse citation searches starting with these two papers. While this literature does not express its results in the PAC setting, and there is no doubt the above works pursue a different computational strategy, I think there are nonetheless some strong similarities. In fact, the above papers go beyond the submitted paper in looking at piecewise polynomial estimates (for smoothness classes beyond piecewise constant), adaptivity to model order, multiple dimensions, all while attaining optimal performance with efficient algorithms. I don't know if those results can be translated to give an apples-to-apples comparison with the submitted work, but if so, I would find that really interesting.Other comments:You say that you are restricting to the unit interval "without loss of generality." Yet, it's not clear whether your technique extends to densities with unbounded support.At line 163, what do you mean that "the L_1 distance is a stronger metric". By Holder's inequality, and since your domain is the unit interval, we know that the L1 norm is bounded by the L2 norm, but wouldn't you need to reverse direction for your statement to make sense? I'm possibly forgetting some basic fact here.The first paragraph of the "preliminaries" section confuses me on one point. You say that "for convenience in the rest of the paper we work with the L_1 distance." Yet you are also allowing distributions with point masses, in which case the distribution does not have a Lebesgue density. If you are allowing point masses, I think you need to stick to TV, and when you say density, be clear about the dominating measure. If you are allowing mixed continuous-discrete distributions, the term "density" seems inappropriate. You're estimating the probability measure.The term "semi-agnostic learning" in the title of subsection 3.2 is not defined.In the last line of the statement of theorem 6, I think you mean q' instead of q.