Rates of convergence for nearest neighbor classification
We analyze the behavior of nearest neighbor classification in metric spaces and provide finite-sample, distribution-dependent rates of convergence under minimal assumptions. These are more general than existing bounds, and enable us, as a by-product, to establish the universal consistency of nearest neighbor in a broader range of data spaces than was previously known. We illustrate our upper and lower bounds by introducing a new smoothness class customized for nearest neighbor classification. We find, for instance, that under the Tsybakov margin condition the convergence rate of nearest neighbor matches recently established lower bounds for nonparametric classification.
This paper develops a new way of analyzing k-Nearest Neighbor prediction for classification problems. Nearest Neighbor prediction is a simple and well studied classification method that, although the first results date decades back, has recently seen some renewed interest and new development. This paper provides a new angle on Nearest Neighbor analysis by incorporating that the behavior of NN classifiers actually automatically adapt to local scaling (with respect to the probability distribution) of the input space. This aspect has not been taken into account in previous studies on NN.Previous analysis of NN has been in terms of smoothness parameters and bounds provided reflect some sort of worst case over the data space of this smoothness parameters. Bound derived like this can be rather loose. This study aims to overcome this drawback. The authors define a new notion of margin (that depends on k and n) and bound the expected error (with repect to the Bayes) in terms of the probability weight of this margin. This is complemented with a lower bound in terms of a similar notion. As a result, the authors obtain a general consistency result as well as some bounds under additional assumptions.The drawback of the paper is that most of the results have a bit of a “in the middle of the analysis” feel (with the exception of Section 2.6). It is not clear how useful it is to say that the error is bounded by the weight of some subset of the domain, when there is no handle/estimate on how much weight there is on that set...However, I do think that this work provides sufficient novel insights on the behavior of NN classifiers to justify publication at NIPS. Moreover, the paper is a nice read.