A Theory-Based Evaluation of Nearest Neighbor Models Put Into Practice
In the k-nearest neighborhood model (k-NN), we are given a set of points P , and we shall answer queries q by returning the k nearest neighbors of q in P according to some metric. This concept is crucial in many areas of data analysis and data processing, e.g., computer vision, document retrieval and machine learning. Many k-NN algorithms have been published and implemented, but often the relation between parameters and accuracy of the computed k-NN is not explicit. We study property testing of k-NN graphs in theory and evaluate it empirically: given a point set P ⊂ R δ and a directed graph G = (P, E), is G a k-NN graph, i.e., every point p ∈ P has outgoing edges to its k nearest neighbors, or is it -far from being a k-NN graph? Here, -far means that one has to change more than an -fraction of the edges in order to make G a k-NN graph. We develop a randomized algorithm with one-sided error that decides this question, i.e., a property tester for the k-NN property, with complexity O( √ nk 2 / 2 ) measured in terms of the number of vertices and edges it inspects, and we prove a lower bound of Ω( n/ k). We evaluate our tester empirically on the k-NN models computed by various algorithms and show that it can be used to detect k-NN models with bad accuracy in significantly less time than the building time of the k-NN model.
SUMMARY: The paper studies the problem of testing whether a graph is epsilon-far from a kNN graph, where epsilon-far means that at least epsilon-fraction of the edges need to be changed in order to make the graph a kNN graph.  The paper presents an algorithm with an upper bound of O(\sqrt{n}*k^2/\epsilon^2) number of edge/vertex queries and a lower bound of \Omega(\sqrt{n}).   COMMENTS:  1. Ln 175. I guess "\omega" should be "p"  2. The result of Proposition 12 is interesting which bounds the number of points that can be in the kNN set of a particular point. The bound is k times the known bound for 1-NN. I wonder if this could be tightened somehow. Also, are there lower bounds for the 1-NN result?  3. Is there a reason why the \epsilon is hidden from the lower bound?  4. It looks like these result don't explicitly show the dependence on the dimension of the space. I guess similar comments about the lower bound: could we get some lower bounds that depend on the dimension of the space? Currently the lower bound is in 1-dimension. I guess we will have some factor that depends exponentially in the dimension of the space.  5. In practice, some mistakes are more costly than others. For example, deleting the edge to the k-th nearest neighbor and adding in an edge to the (k+1)-th nearest neighbor is not as bad as deleting the edge to the k-th nearest neighbor and adding an edge to the farthest point. Can the analysis be extended to take this into account and e.g. have some weighted cost?   OVERALL: Testing whether a graph is approximately a kNN graph is an interesting problem with implications to approximate nearest neighbor (ANN) procedures. The paper provides an algorithm which tests whether such an ANN index actually approximates the exact kNN method which has reasonable computational cost which could have applications in cheaply testing whether an ANN index is acceptable for usage. The algorithm and theoretical results presented are to my knowledge novel. These results come with some experimental results on a number of datasets which suggest that this may have relevence in practice.