Non-metric Similarity Graphs for Maximum Inner Product Search
In this paper we address the problem of Maximum Inner Product Search (MIPS) that is currently the computational bottleneck in a large number of machine learning applications. While being similar to the nearest neighbor search (NNS), the MIPS problem was shown to be more challenging, as the inner product is not a proper metric function. We propose to solve the MIPS problem with the usage of similarity graphs, i.e., graphs where each vertex is connected to the vertices that are the most similar in terms of some similarity function. Originally, the framework of similarity graphs was proposed for metric spaces and in this paper we naturally extend it to the non-metric MIPS scenario. We demonstrate that, unlike existing approaches, similarity graphs do not require any data transformation to reduce MIPS to the NNS problem and should be used for the original data. Moreover, we explain why such a reduction is detrimental for similarity graphs. By an extensive comparison to the existing approaches, we show that the proposed method is a game-changer in terms of the runtime/accuracy trade-off for the MIPS problem.
The authors identify the issue in solving MIPS by reducing it to NNS and propose the use of non-metric similarity graphs to solve MIPS in its native form. While the idea and the empirical results are extremely promising, there are a couple of issues that temper my enthusiasm for this manuscript significantly.    Major: - While the theoretical results claim the correctness of the Greedy Walk algorithm (Alg 1) with s-Delaunay graphs, it is hard for me to understand (concretely or even intuitively) how the approximation in the graph construction (so that it is no longer a s-Delaunay graph) affects the accuracy of Alg 1 for MIPS. With existing methods such as LSH/RP-trees+reduction or Greedy-MIPS, the algorithm will find the exact answer(or within some approximation) eventually or drive the probability of not finding the exact answer to really low values (with RP-trees providing stronger guarantees than LSH). However, with Alg 1, there is no guarantee on the approximation of the obtained answer. - Empirical results in terms of actual runtimes are a great way of evaluating the proposed solution. However, it makes it hard for readers to judge whether the improvements of the proposed solution because of the novel/proposed algorithm (which is discussed in detail in the manuscript) or because of implementation details such as programming languages/libraries used, any kind of cache locality/multithreading/register level optimizations used, etc (which are not discussed/elaborated/focused on in the manuscript and hence unknown to the reader). And these implementation details can be quite significant when we are discussing runtimes in the order of milliseconds (unless these numbers are generated by repeating the runs multiple times and ensuring there are no competing programs on the evaluation machine). An accompanying empirical comparison using number of vector operations (for example, dot products) as a surrogate for time would provide a fairer comparison of the algorithms being evaluated, and make the gains from the proposed algorithm more clear.   Minor: - How are the recall vs. time curves generated for the proposed graph based solution in Figures 2,3? What parameters of the algorithm are being varied to span the complete speed-accuracy tradeoff? This is not clear to me from the description of Alg 1 and Section 4. - As per lines 41-42, the authors claim to provide "theoretical analysis to justify the use of similarity graphs". However, I couldn't find any concrete theoretical result supporting this claim (unless I am missing something). - How does the value of the parameters M (graph vertex degree) and L (the size of the priority queue maintained during the search) affect the performance of the proposed MIPS solution? Is the proposed method sensitive to these parameters? How dependent are these parameters on the dataset?  ================ Post author-response edit: I want to thank the authors for their thoughtful responses; I acknowledge that I have read them. The authors have addressed two of my major concerns and I am thoroughly satisfied with one. However, the other concern (re: approximation-accuracy discussion), I appreciate the explanation and intuition -- it definitely helps me understand things better, but it still lacks certain amount of rigour, and the fact that there is no recovery from a local minima (or alternately, no guarantees on the local minima relative to the global minima) still leaves me somewhat unsatisfied. However, my score is still fairly positive so I will leave it at that.