Robust Kernel Density Estimation by Scaling and Projection in Hilbert Space
While robust parameter estimation has been well studied in parametric density estimation, there has been little investigation into robust density estimation in the nonparametric setting. We present a robust version of the popular kernel density estimator (KDE). As with other estimators, a robust version of the KDE is useful since sample contamination is a common issue with datasets. What "robustness" means for a nonparametric density estimate is not straightforward and is a topic we explore in this paper. To construct a robust KDE we scale the traditional KDE and project it to its nearest weighted KDE in the L 2 norm. This yields a scaled and projected KDE (SPKDE). Because the squared L 2 norm penalizes point-wise errors superlinearly this causes the weighted KDE to allocate more weight to high density regions. We demonstrate the robustness of the SPKDE with numerical experiments and a consistency result which shows that asymptotically the SPKDE recovers the uncontaminated density under sufficient conditions on the contamination.
Summary:This paper considers the problem of robust kernel density estimation, where the authors assume that most observations come from a target density f_tar, but some observations are drawn from a contaminating density f_con.  They consider the setting that the observation density f_obs is a known convex combination of f_tar and f_con, but we do not know f_tar or f_con. In practice, the convex combination is not known, but can be treated as a tuning parameter, to adjust between more and less robust estimates. They also have "Assumption A", which states that the contamination density f_con is uniform at r on the support of the target density f_tar, but is less than r otherwise.  Under this assumption, they present a decontamination operator that maps a contaminated density to the target density.  The decontamination operator rescales the observation density f_obs by a certain amount, then shifts it down, pointwise, to make the resulting function integrate to 1.  This addresses the infinite sample, or population version of the problem.  To apply this a finite sample, we could first approximate the observation density f_obs by a classic kernel density estimator, and then apply the decontamination operator.  However, this does not work in practice, because computing the amount of the shift is not computationally tractable.  Instead, they propose taking the L2 projection of the rescaled function onto the convex hull of kernel functions evaluated at each of the sample points.  Finding this projection amounts to a quadratic program involving the Gram matrix of L2 inner products of the kernel functions centered at each of the sample points, which can be computed efficiently for many kernels of interest.  They eventually prove that the L1 distance between the "SPKDE" estimator and the target density converges to 0 in probability. The authors say that the convergence of their estimator to f_tar, under sufficient conditions, is the first result of its type. In their experiments, they first compare the performance of SPKDE to traditional KDE on artificially generated data from distributions satisfying Assumption 1.  SPKDE clearly outperforms KDE. In the experiments on real data, the estimated density based on contaminated training data is compared to an estimated density based on uncontaminated [I assume uncontaminated -- this wasn't entirely explicit] test data.  The comparison was done using KL-divergence in both directions. The Wilcoxon signed rank test is used to see if the performance differences are significant across the N datasets.General Comments:The paper was quite interesting and readable.  I just have a few suggestions.  First, density estimation on R is so easy to visualize, I feel like you could do a better job of illustrating the differences between SPKDE, KDE, rejKDE, and RKDE.  You could pick synthetic datasets that play to the strengths of each.  I think it would be more impactful and interesting.On the real datasets, using KL divergence in both directions seems reasonable, but is a bit heavy, since you're measuring divergences with respect to another KDE estimate on the test data. [I've assumed that this test data is uncontaminated, but this could be explained more clearly.]  What about using test likelihood as another performance measure?  This seems very standard for measuring the quality of a distribution's fit to data, and there are no issues of symmetry or having to first "fit" the test data.With regard to RKDE performing better in one direction of the KL divergence, and the SPKDE performing better in the other direction: can this difference be visualized?  Under what assumptions does RKDE perform best, and how do RKDE and SPKDE compare in that situation, and vice versa?Specific Comments:- Seems like Proposition 2 is essentially proved by line 136 on page 3.  Lemma 1 seems unnecessary to prove Proposition 2.  Maybe state Proposition 2 before Lemma 1. And right after say that the transform used "is well-defined more generally" and then give Lemma 1?  Or even postpone Lemma 1 until you actually need it in Section 3?  As it is currently presented, it's seems like Proposition 2 depends on Lemma 1 (which has a long proof), when really it's quite simple (unless I've misunderstood).- The acronym SPKDE isn't formally introduced before it's used in the abstract or the introduction.- Justification for uniformity assumption on f_con: It's unclear what aspects of Assumption A are being supported by the second paragraph on page 3 (starting line 113).  Assumption A has two parts: 1) f_con is uniform on the support of f_tar, and 2) The values of f_con outside of supp(f_tar) is bounded above by its value on supp(f_tar).  It seems like both of these are important to the method.- Line 111 on page 3, I think you mean that the union is exactly all densities whose _support_ has finite Lebesgue measure?- The differences between what's happening in Figure 2 and Figure 3 are pretty subtle.  Might be there a more clear illustration of the differences?- It's rather difficult to find any of these contamination models particularly "natural" or "reasonable".  However, on line 182 of page 4, you characterize the assumption in terms of the type of target densities that can be recovered.  At that point, it might be worth reminding the reader of the comment on line 111, that any target density with finite support is recoverable under Assumption A your method.