In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations. We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views. Formally, multiple views of the same sentence are mapped to close representations. On the contrary, views from other sentences are mapped further. By contrasting different linguistic views, we aim at building embeddings which better capture semantic and are less sensitive to the sentence outward form.
Overview: This work proposes to learn sentence embeddings using both contrastive learning and multiple "views" of sentences.  This work largely builds off of [1], including using the same objective, but uses a multi-view approach to modeling.
    - They apply the concept of multi-view models, specifically combining tree and linear LSTMs to learning sentence representations.
    - They prepare a new, large-scale book dataset, which is useful because the previously commonly used book dataset was taken down for legal reason.
    - They provide a fairly broad set of analyses on their model, both quantitative and qualitative, performance-driven and analysis-driven.
- Review: The ideas and models presented in this paper are not new, while the supporting experiments are not very well done or convincing. Overall, I recommend rejecting this work.
- The models are contrastively learned in that they are trained to embed "similar" sentences nearby in the embedding space, and "dissimilar" sentence far away, where "similar" sentences are defined as consecutive sentences. This method of learning textual representations is well-established in the NLP literature, mostly prominently in recent years with word embedding models like Skip-Gram and in sentence embedding models like in [1], [2], [3] (the next sentence prediction task), and several more.
- In practice, the multiple views of each sentence that this paper considers boils down to encoding the sentence with a bidirectional LSTM and a TreeLSTM and concatenating the representations from each encoder. This idea again has been established in the literature ([4], [5], [6]).
- The experiments don't seem setup to demonstrate that the multiple views are beneficial over a single view. In Table 1, there are rows for just an LSTM or just a TreeLSTM, but they seem to be trained with labeled data whereas the proposed method is trained self-supervised. A more informative comparison to demonstrate the value of using multiple views would be to train the LSTM and TreeLSTM with the same objective (and ideally model size). Overall, I don't think the claims in the paper are well-supported by the model proposed or the experiments.
- I have a number of concerns about the experiments.
    - "Models are trained on a single epoch on the entire corpus without any train-test split": so there is no early stopping? Why stop training after one epoch? Was there any indication you were overfitting the data?
    - "The training phase was stopped after 33 hours of training": Why stop there? Computational constraints? Later comments suggest this is quite premature ("training phase was completed on only 4.6M sentences among the 78M available").
    - The results seem to indicate that this method underperforming recent work significantly.
Areas of improvement
- Some of the language in the introduction and conclusion are a bit of a stretch. Using a linear and tree LSTM (based on dependency parses) doesn't really represent a "diversity of linguistic structures". 
- Related work: There's no mention of pretained language models, which could be seen as a form of representation learning for language, and have been hugely impactful in NLP.
- Method
    - Missing negative in the log likelihood
    - Why do you use inner product if other works "report excellent results" with other scoring functions?
    - "assumes the underlying structure of the sentence to be a sequence, while allowing for long term dependencies": If anything, the treeLSTM more easily allows for long-term dependencies than the linear LSTM.
    - "Negative examples are obtained using the dependency Tree LSTM": I'm not totally sure how the negatives are obtained here.
    - "The target sequence is encoded using the sequential Tree LSTM, while the positive and negative samples are encoded using the ChildSum Tree LSTM": why are the sentences not all encoded with the same encoder?
    - It looks really odd that most of Table 1 is empty. Given your model, I imagine it can't have been that difficult to evaluate more baselines (BiLSTM and TreeLSTM) on the rest of the tasks.
    - It'd be nice if you could clearly indicate in Table 1 which method is yours.
- Results and Analysis
    - The standard evaluation setting for sentence embeddings would be GLUE or SuperGLUE.
    - A glaringly missing baseline is BERT (or any of its relatives), which is also self-supervised.
    - The results are underwhelming, and as the author admits, somewhat premature as training didn't seem to finish.
    - 5.2: what are the contrastive LSTM and Tree LSTM? Are those the learned encoders from the "Contrastive Tree" in Table 1, or are they trained from scratch?
    - I don't think the analyses in Sections 5.2 and 5.3 or Figure 2 are particularly useful.
- There are a noticeable number of typos. For example, in the abstract: "this linguist[ic] diversity" and "better capture semantic[s]". It'd be worthwhile to look over the paper closely for typos.
[1] AN EFFICIENT FRAMEWORK FOR LEARNING SENTENCE REPRESENTATIONS. Lajanugen Logeswaran and Honglak Lee
[2] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. Yacine Jernite, Samuel R. Bowman, David Sontag
[3] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
[4] Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang.
[5] Enhanced LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen.
[6] Improving Sentence Representations with Consensus Maximisation. Shuai Tang, Virginia R. de Sa.