The meaning of a sentence is a function of the relations that hold between its words. We instantiate this relational view of semantics in a series of neural models based on variants of relation networks (RNs) which represent a set of objects (for us, words forming a sentence) in terms of representations of pairs of objects. We propose two extensions to the basic RN model for natural language. First, building on the intuition that not all word pairs are equally informative about the meaning of a sentence, we use constraints based on both supervised and unsupervised dependency syntax to control which relations influence the representation. Second, since higher-order relations are poorly captured by a sum of pairwise relations, we use a recurrent extension of RNs to propagate information so as to form representations of higher order relations. Experiments on sentence classification, sentence pair classification, and machine translation reveal that, while basic RNs are only modestly effective for sentence representation, recurrent RNs with latent syntax are a reliably powerful representational device.
The paper presents an extension of relation networks (RNs) for natural language processing. RNs are designed to represent a set as a function of the representations of their elements. This paper treats a sentence as a set of words. Whereas regular RNs assume that the representation of a set is a uniform aggregation of the representation of the pairs of elements in the set, this paper proposes to weight the relevance of the pairs according to their tree-structured dependency relations between the words. The authors evaluate on a suite of NLP tasks, including SNLI, Quora duplicate question ID, and machine translation. They show marginal improvements over naive baselines, and no improvement over SOTA.
I am concerned about both the motivation for and the novelty of this work. My reading of this work is that the authors try to reverse engineer a TreeRNN in terms of RNs, but I am not sure what the reason is for wanting to use the RN framework in order to derive an architecture that, IIUC, essentially already exists. I can't find any fundamentally meaningful differences between the proposed architecture and the existing work on TreeRNNs, and the results suggest that there is nothing groundbreaking being proposed here. It is possible I am missing some key insight, but I do believe the burden is on the authors to highlight where the novelty is. The intro *and* related work sections should both be rewritten to answer the question: what is the insufficiency with current sentence encoding models that is addressed by this architecture? Currently, the intro addresses the tangential question: what is the insufficiency with RNs for NLP that is addressed by this architecture? If the latter is the question the authors want to answer, they need to first answer: why should we want to cast sentence encoders as RNs as opposed to any of the (many) other available architectures? Without a firmer understanding of what this paper contributes and why, I can't recommend acceptance. More detailed comments for the authors below. 
- You introduce a few naive baselines, but none of these is a TreeRNN. TreeRNNs are the obvious baseline, and you should be comparing on each and every evaluation task, even if there is no previously published result for using tree RNNs on that task. For the one result (SNLI, table 1) on which there is previous work using TreeRNNs, the table confirms my intuition that the proposed model is no improvement over the TreeRNN architecture. It seems very important to address this comparison across all of the evaluation tasks.
- I like the notion of marginalizing over latent tree structures, but the related work section needs to make clear what is being contributed here that is different from the cited past work on this problem
- On the MT eval, why are you missing values for zh-en on the NMT models that are actually competitive? I think many of these models are open-source or easy to reimplement? Its hard to draw conclusions when from such a gappy table.
- Only table 2 has significance values (over naive baseline that is) which implies that the other results are not significant? That is disconcerting. 
- I am disappointed in the analysis section. As is, you provide an ad-hoc inspection of some inferred trees. I find this odd since there is no evidence that the tree-ness of the architecture (as opposed to, e.g., recurrence or attention) is what leads to quantitative improvements (at least according to the experimental results in the tables), so there is no reason we should actually expect the trees to be good or interesting. My interpretation of these cherry-picked examples is that the learning is fighting the architecture a bit, basically "learning a tree" that reduces to being an attention mechanism that up-weights one or two salient words. 
- The analysis I *wanted* to see instead was why recursion helped for sentence classification, it did not for MT. You give an intuition for this result but no evidence. (That is assuming that, quantitatively, this trend actually holds. Which maybe is not the case if none of the results are significant.)
- In general, regarding evaluation, SNLI is overfit. You should use MNLI at least. I have trouble interpreting progress on SNLI as "actual" progress on language representation.
- The related work section as a whole is too short. If you need to cut space, move technical content to appendix, but don't compromise in related work. You listed many relevant citations, but you have no context to situate your contribution relative to this past work. What is the same/different about your method? You should provide an answer to that for each and every paper you cite.