k ⇤ -Nearest Neighbors: From Global to Local
The weighted k-nearest neighbors algorithm is one of the most fundamental nonparametric methods in pattern recognition and machine learning. The question of setting the optimal number of neighbors as well as the optimal weights has received much attention throughout the years, nevertheless this problem seems to have remained unsettled. In this paper we offer a simple approach to locally weighted regression/classification, where we make the bias-variance tradeoff explicit. Our formulation enables us to phrase a notion of optimal weights, and to efficiently find these weights as well as the optimal number of neighbors efficiently and adaptively, for each data point whose value we wish to estimate. The applicability of our approach is demonstrated on several datasets, showing superior performance over standard locally weighted methods.
This paper proposes a reweighting method of nearest neighbors (NNs) for NN classification and regression. The authors derive the upper bound of the difference between the model output and the target value, and the minimization of the upper bound is obtained by a simple and quick iterative process. As a result, the weights \alpha for k nearest neighbors are determined for given data, as well as the optimal number k.
The mathematical derivation in this work is novel and interesting, but the writing can be improved. The derivation uses Hoeffding’s inequality for noise on labels, with a mild assumption on the maximum noise. The analysis using KKT condition is straightforward, and it results in an interesting strategy for finding weights.  I want to support the acceptance of this paper, but I have several concerns: 1) According to the derivation, the upper bound of the risk (P1) is given as C\lambda (line 112). With some simple calculations using appropriate parameters, this upper bound does not give a small value. For example, with b=1, delta=0.1, and k*=10, and \lambda greater than the minimum (\beta_i + 1/k*) from Eq.(4), the minimum upper bound I roughly calculated is about 0.24. For example in classification, this does not seem to be a meaningful upper bound. Can I reduce the upper bound further with appropriate k* (k*=3-10)?  2) In high dimensional space, distance to the nearest neighbors is unreliable in many cases. In particular, the distance is highly concentrated, and the nearest neighbor does not appear very closely to the test point. How do the authors advocate the Lipschitz condition that the function deviation is bounded linearly by the distance within the range where the nearest neighbors appear?  3) Though the derivation and the result is interesting, the paper is not well-written. The structure and the writing can be improved.  4) In the discussion from line 170 to 179, the two presented examples are redundant. The first example is the special case of the second example.  5) The title has to be changed. From the title, I cannot have any information about this paper.  The derivation in this manuscript is new, and I like the way of obtaining the proposed optimization procedure. However, further honest analysis will make this paper more interesting when this method is useful and when it is not. Also, if the authors can provide some guarantee that the writing can be significantly improved, I want to recommend the acceptance of this paper.