Zero-training Sentence Embedding via Orthogonal Basis
We propose a simple and robust training-free approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is its novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace. Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representation. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.
 Paper overview: This paper proposes a new geometry-based method for sentence embedding from word embedding vectors, inspired by Arora et al (2017). The idea is to quantify the novelty,significance and corpus-wise uniqueness of each word. In order to do so, they analyze geometrically how the word vector of the target word relates to 1) the subspace created by the word-vectors in its context 2) its alignment with the meanings in its context (using SVD) 3) its presence in the all the corpus. For each of these aspects, they output a score or weight. The final sentence representation is a weighted average, using these scores, of the word vectors of the sentence. Remarks and questions: 1) In table 1, Glove and word2vec are word representations, how is the sentence representation computed here? 2) The authors are not comparing to what is now considered the state of the art methods, such as Quick thoughts vectors (ICLR 2018, *an efficient framework for learning sentence representations* by Logeswaran et al.), Transformer (Attention is all you need by Vaswani et al.) and ELMo (Deep contextualized word representations, by Peters et al.). Points in favor: 1) Results: The method gives the best performance for non-training methods with an +2 point improvement on average, although it cannot beat training methods (see Table 3, for instance). 2) On the result tables, it should be reported also the std, not just the average, so the reader can evaluate if the difference between the methods is statistically significant. 3) Inference speed: the method is fast (see table 5) 4) stability of the results: The method is robust to slight changes in the hyperparameters such as the size of the window, number of principal components used, etc (see Fig 2) Points against: The methods presented in the paper are not novel. The main novelties are the geometrical analysis on the contribution of each word of the sentence to the sentence overall semantic meaning, and the definition of the scores (eqs 4,6,8) that allow to improve the weighted average sentence representation (eq 9), an idea already present in Arora et al.*s paper. Conclusion: Although the geometric analysis of the paper is interesting, I dont think it is sufficient to justify a paper at ICLR, unless, after comparison with the other methods proposed previously, the proposed model is still competitive and the difference is statistically significant.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Zero-training Sentence Embedding via Orthogonal Basis
We propose a simple and robust training-free approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is its novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace. Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representation. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Paper overview: This paper proposes a new geometry-based method for sentence embedding from word embedding vectors, inspired by Arora et al (2017). The idea is to quantify the novelty,significance and corpus-wise uniqueness of each word. In order to do so, they analyze geometrically how the word vector of the target word relates to 1) the subspace created by the word-vectors in its context 2) its alignment with the meanings in its context (using SVD) 3) its presence in the all the corpus. For each of these aspects, they output a score or weight. The final sentence representation is a weighted average, using these scores, of the word vectors of the sentence. Remarks and questions: 1) In table 1, Glove and word2vec are word representations, how is the sentence representation computed here? 2) The authors are not comparing to what is now considered the state of the art methods, such as Quick thoughts vectors (ICLR 2018, *an efficient framework for learning sentence representations* by Logeswaran et al.), Transformer (Attention is all you need by Vaswani et al.) and ELMo (Deep contextualized word representations, by Peters et al.). Points in favor: 1) Results: The method gives the best performance for non-training methods with an +2 point improvement on average, although it cannot beat training methods (see Table 3, for instance). 2) On the result tables, it should be reported also the std, not just the average, so the reader can evaluate if the difference between the methods is statistically significant. 3) Inference speed: the method is fast (see table 5) 4) stability of the results: The method is robust to slight changes in the hyperparameters such as the size of the window, number of principal components used, etc (see Fig 2) Points against: The methods presented in the paper are not novel. The main novelties are the geometrical analysis on the contribution of each word of the sentence to the sentence overall semantic meaning, and the definition of the scores (eqs 4,6,8) that allow to improve the weighted average sentence representation (eq 9), an idea already present in Arora et al.*s paper. Conclusion: Although the geometric analysis of the paper is interesting, I dont think it is sufficient to justify a paper at ICLR, unless, after comparison with the other methods proposed previously, the proposed model is still competitive and the difference is statistically significant.
assistant
 Paper overview: This paper proposes a new geometry-based method for sentence embedding from word embedding vectors, inspired by Arora et al (2017). The idea is to quantify the novelty,significance and corpus-wise uniqueness of each word. In order to do so, they analyze geometrically how the word vector of the target word relates to 1) the subspace created by the word-vectors in its context 2) its alignment with the meanings in its context (using SVD) 3) its presence in the all the corpus. For each of these aspects, they output a score or weight. The final sentence representation is a weighted average, using these scores, of the word vectors of the sentence. Remarks and questions: 1) In table 1, Glove and word2vec are word representations, how is the sentence representation computed here? 2) The authors are not comparing to what is now considered the state of the art methods, such as Quick thoughts vectors (ICLR 2018, *an efficient framework for learning sentence representations* by Logeswaran et al. ), Transformer (Attention is all you need by Vaswani et al.) and ELMo (Deep contextualized word representations, by Peters et al.). Points in favor: 1) Results: The method gives the best performance for non-training methods with an +2 point improvement on average, although it cannot beat training methods (see Table 3, for instance). 2) On the result tables, it should be reported also the std, not just the average, so the reader can evaluate if the difference between the methods is statistically significant. 3) Inference speed: the method is fast (see table 5) 4) stability of the results: The method is robust to slight changes in the hyperparameters such as the size of the window, number of principal components used, etc (see Fig 2) Points against: The methods presented in the paper are not novel. The main novelties are the geometrical analysis on the contribution of each word of the sentence to the sentence overall semantic meaning, and the definition of the scores (eqs 4,6,8) that allow to improve the weighted average sentence representation (eq 9), an idea already present in Arora et al. *s paper. Conclusion: Although the geometric analysis of the paper is interesting, I dont think it is sufficient to justify a paper at ICLR, unless, after comparison with the other methods proposed previously, the proposed model is still competitive and the difference is statistically significant.