Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks.
After reading through the authors' comments and rereading parts of the submission, I have become a little more positive about this paper. 
I am still unsure about the contributions to the ICLR community. The authors merely state "We believe our contributions to ICLR community are clear and valuable" without backing up this claim with an argument. 
But in the rest of the authors' comment, they make some good points. I think those points should be made more prominently in the paper itself. I would suggest that the authors describe their approach as using different, complementary encoders of the input sentence and consensus maximization. If they wish to describe this as multi-view learning, that's fine, but I think using the term "consensus maximization" (or something more descriptive like that) in prominent places would be helpful. 
If the approach is applicable beyond sentence embedding learning, then it would behoove the authors to describe the approach in a general way so that readers will see how they can apply it to their own tasks. As currently written, the paper is very much focused on sentence embedding learning, which causes me to think that the paper is more appropriate for an NLP venue. But it is true that ICLR publishes papers that are application-specific, so I can't consider this to be a deal-breaker for the paper.
I raised my score to a 6.
--------------------------------- original review follows: ----------------------------------
This paper describes experiments in learning sentence embeddings from unlabeled text. The paper compares a few different compositional architectures and training objectives. The story of the paper focuses on the training of multiple architectures jointly for a single sentence, then ensembling those architectures at test time to represent sentences. One architecture is an RNN and the other is a word averaging model, and the idea is that these two architectures capture different "views" of the sentence. 
Pros:
As a general-purpose method to get sentence embeddings without using any resources other than unannotated text documents, this approach has strong results, including solid performance on the SentEval tasks and relatively-low training times. 
It was also nice to see how the results depend on the domain of the training data. Review data definitely helps on the several sentiment-related tasks, which provides further evidence of a worrisome aspect of SentEval. 
Cons:
Overall, the paper feels incremental and is likely a better fit for an NLP conference. What are the generalizable contributions to the ICLR community? Given the known differences between RNNs and word averaging models for sentences (especially on the SentEval tasks, which, as the authors note, was discussed by Hill et al.), it's entirely unsurprising that combining the two would be a good idea. But even if this were not the case, the ubiquity of ensembles outperforming single models in deep learning also makes it unsurprising that combining these two kinds of model architectures would be beneficial. So I'm just not sure if there is a significant contribution beyond the NLP results. These results seem solid (though a bit incremental), but if the primary contribution is empirical, then the paper would be a better fit for an NLP venue. 
In addition, I'm not sure if "multi-view" is an appropriate description of the approach. In Sec. 1, we find the sentence "Compared to earlier work on multi-view learning (de Sa, 1993; Dhillon et al., 2011) that takes data from various sources or splits data into disjoint populations, our framework processes the exact same input data in two distinctive ways."  Therefore, maybe it's not quite accurate to describe this approach using the term "multi-view learning"? I think it would make more sense to use a different term rather than stretch the definition for a well-known one. 
I kept expecting the paper to present results when combining the generative and discriminative objectives, but as far as I can tell, this was never done. What would happen if one were to use multitask learning and just optimize the sum of the two losses? 
I'd suggest citing and comparing to the results from "Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning" by Subramanian et al. (ICLR 2018) and the Byte mLSTM from "Learning to generate reviews and discovering sentiment" by Radford et al. 
I'm not sure how excited we should get about not using any annotations or structured resources for learning sentence embeddings. The authors do not motivate this goal. 
Below are more specific comments/questions:
Sec. 2.2 contains the sentence "Ideally, the inverse of h should be easy to compute so that during testing we can set g = h^-1." At this point in the paper, it is not clear what g is going to be applied to at test time, since presumably the following sentence is not going to be available at test time, right? I think it would be good to discuss how the model is going to be used at test time before discussing the inverse of h. 
Sec. 2.3: 
In Eq. (3), why does the denominator sum always start at 1 no matter what i and j are? That is, why would the denominator always sum over the first N sentences in the dataset?
I think the pooling methods in Table 2 should be described in Section 2. 
In Table 2, it is not clear what h_i^{M_i} is. If M_i is the number of words in sentence i, that should be mentioned somewhere. 
Sec. 3.1:
"For a given sentence input s with M words, suggested by (Pennington et al., 2014; Levy et al., 2015), the representation is calculated as z = (\hat{z}_f + \hat{z}_g) / 2, where \hat{z} refers to the post-processed and normalised vector, and is mentioned in Table 2."  I don't understand. Where is this mentioned in Table 2?
Minor issues follow:
Sec. 1:
"Distributional hypothesis" --> "the distributional hypothesis"
"in machine learning community" --> "in the machine learning community"
"and distributional hypothesis" --> "and the distributional hypothesis"
"the linear/log-linear models" --> "linear/log-linear models"
"based on distributional hypothesis" --> "based on the distributional hypothesis"
"contraint" --> "constraint"
Sec. 2:
"marry RNN-based sentence encoder" --> "marry RNN-based sentence encoders" or "marry the RNN-based sentence encoder"
Sec. 2.1:
"only hidden state" --> "only the hidden state"
Sec. 2.2:
"prior work with generative objective" --> "prior work with generative objectives"
Sec. 2.3:
"with discriminative objective learns" --> "with the discriminative objective learns"
Sec. 3.1:
In Table 3, I don't see where superscript 5 is shown in the table.
Sec. 3.2:
In Table 5, the numbered superscripts at the top of the table do not show up next to the methods in the actual table rows.