We propose a simple and robust non-parameterized approach for building sentence representations. Inspired by the Gram-Schmidt Process in geometric theory, we build an orthogonal basis of the subspace spanned by a word and its surrounding context in a sentence. We model the semantic meaning of a word in a sentence based on two aspects. One is its relatedness to the word vector subspace already spanned by its contextual words. The other is the word's novel semantic meaning which shall be introduced as a new basis vector perpendicular to this existing subspace. Following this motivation, we develop an innovative method based on orthogonal basis to combine pre-trained word embeddings into sentence representations. This approach requires zero training and zero parameters, along with efficient inference performance. We evaluate our approach on 11 downstream NLP tasks. Experimental results show that our model outperforms all existing zero-training alternatives in all the tasks and it is competitive to other approaches relying on either large amounts of labelled data or prolonged training time.
Paper overview: This paper proposes a new geometry-based method for sentence embedding from word embedding vectors, inspired by Arora et al (2017). The idea is to quantify the novelty,significance and corpus-wise uniqueness of each word. In order to do so, they analyze geometrically how the word vector of the target word relates to 1) the subspace created by the word-vectors in its context 2) its alignment with the meanings in its context (using SVD) 3) its presence in the all the corpus. For each of these aspects, they output a score or weight. The final sentence representation is a weighted average, using these scores, of the word vectors of the sentence.  
Remarks and questions: 
     1) In table 1, Glove and word2vec are word representations, how is the sentence representation computed here? 
     2) The authors are not comparing to what is now considered the state of the art methods, such as Quick thoughts vectors (ICLR 2018, 'an efficient framework for learning sentence representations' by Logeswaran et al.), Transformer (Attention is all you need by Vaswani et al.) and ELMo (Deep contextualized word representations, by Peters et al.). 
Points in favor:
    1) Results: The method gives the best performance for non-training methods with an +2 point improvement on average, although it cannot beat training methods (see Table 3, for instance). 
   2) On the result tables, it should be reported also the std, not just the average, so the reader can evaluate if the difference between the methods is statistically significant.
    3) Inference speed: the method is fast (see table 5) 
    4) stability of the results: The method is robust to slight changes in the hyperparameters such as the size of the window, number of principal components used, etc (see Fig 2)
Points against: 
     The methods presented in the paper are not novel. The main novelties are the geometrical analysis on the contribution of each word of the sentence to the sentence overall semantic meaning, and the definition of the scores (eqs 4,6,8) that allow to improve the weighted average sentence representation (eq 9), an idea already present in Arora et al.'s paper. 
 Conclusion: 
     Although the geometric analysis of the paper is interesting, I dont think it is sufficient to justify a paper at ICLR, unless, after comparison with the other methods proposed previously, the proposed model is still competitive and the difference is statistically significant.