Situating Sentence Embedders with Nearest Neighbor Overlap
As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes. We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner. N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs* nearest neighbors. We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures.
The paper provides a method/metric for comparing sentence embedders, based on a nearest neighbor analysis. The method is straightforward: sample a sentence, embed the sentence and lots of sentences from a corpus, find the k nearest neighbors in the corpus to the sampled sentence, do the same for another embedder, calculate the overlap of the two sets of nearest neighbors. The paper is commendably clear and easy to read. The main problems I have with the paper are twofold: 1) it*s not clear this is enough of a contribution for a top-tier ML conference; and 2) it*s not clear what I do with the results. I think the idea of analysing embedders in this way is potentially interesting, but it feels like the paper needs more. This analysis could be a great section in another paper (e.g. one which proposes another embedding method); or perhaps it could be extended in some way, so that the current content only takes up 1/2 the space, and then there*s 1/2 the paper showing how useful this analysis is, for e.g. building better embedders for particular tasks. The abstract starts by talking about how embedders have been evaluated using various benchmarks, and hints at the idea that this new comparative approach could be an alternative. But the new method can*t really be used for evaluation: I don*t come away from the paper knowing whether embedding method A is better than method B, only that A is more like B than C. I think the problem with the paper as it stands is neatly summed up in the conclusion of the paper, which isn*t a conclusion at all: it*s just a mini-abstract. I*d like to know what readers should take away from the results, so that they can potentially build better embedders. I*ve given the paper a 1. rating only because I really don*t think it*s ready for a full ICLR paper, not because I think the method is uninteresting or useless. On the contrary, with some more work and thought about how the analysis could be used, this could be a potentially useful tool.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Situating Sentence Embedders with Nearest Neighbor Overlap
As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes. We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner. N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs* nearest neighbors. We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper provides a method/metric for comparing sentence embedders, based on a nearest neighbor analysis. The method is straightforward: sample a sentence, embed the sentence and lots of sentences from a corpus, find the k nearest neighbors in the corpus to the sampled sentence, do the same for another embedder, calculate the overlap of the two sets of nearest neighbors. The paper is commendably clear and easy to read. The main problems I have with the paper are twofold: 1) it*s not clear this is enough of a contribution for a top-tier ML conference; and 2) it*s not clear what I do with the results. I think the idea of analysing embedders in this way is potentially interesting, but it feels like the paper needs more. This analysis could be a great section in another paper (e.g. one which proposes another embedding method); or perhaps it could be extended in some way, so that the current content only takes up 1/2 the space, and then there*s 1/2 the paper showing how useful this analysis is, for e.g. building better embedders for particular tasks. The abstract starts by talking about how embedders have been evaluated using various benchmarks, and hints at the idea that this new comparative approach could be an alternative. But the new method can*t really be used for evaluation: I don*t come away from the paper knowing whether embedding method A is better than method B, only that A is more like B than C. I think the problem with the paper as it stands is neatly summed up in the conclusion of the paper, which isn*t a conclusion at all: it*s just a mini-abstract. I*d like to know what readers should take away from the results, so that they can potentially build better embedders. I*ve given the paper a 1. rating only because I really don*t think it*s ready for a full ICLR paper, not because I think the method is uninteresting or useless. On the contrary, with some more work and thought about how the analysis could be used, this could be a potentially useful tool.
assistant
The paper provides a method/metric for comparing sentence embedders, based on a nearest neighbor analysis. The method is straightforward: sample a sentence, embed the sentence and lots of sentences from a corpus, find the k nearest neighbors in the corpus to the sampled sentence, do the same for another embedder, calculate the overlap of the two sets of nearest neighbors. The paper is commendably clear and easy to read. The main problems I have with the paper are twofold: 1) it*s not clear this is enough of a contribution for a top-tier ML conference; and 2) it*s not clear what I do with the results. I think the idea of analysing embedders in this way is potentially interesting, but it feels like the paper needs more. This analysis could be a great section in another paper (e.g. one which proposes another embedding method); or perhaps it could be extended in some way, so that the current content only takes up 1/2 the space, and then there*s 1/2 the paper showing how useful this analysis is, for e.g. building better embedders for particular tasks. The abstract starts by talking about how embedders have been evaluated using various benchmarks, and hints at the idea that this new comparative approach could be an alternative. But the new method can*t really be used for evaluation: I don*t come away from the paper knowing whether embedding method A is better than method B, only that A is more like B than C. I think the problem with the paper as it stands is neatly summed up in the conclusion of the paper, which isn*t a conclusion at all: it*s just a mini-abstract. I*d like to know what readers should take away from the results, so that they can potentially build better embedders. I*ve given the paper a 1. rating only because I really don*t think it*s ready for a full ICLR paper, not because I think the method is uninteresting or useless. On the contrary, with some more work and thought about how the analysis could be used, this could be a potentially useful tool.