To construct representations for natural language sequences, information from two main sources needs to be captured: (i) semantic meaning of individual words, and (ii) their compositionality. These two types of information are usually represented in the form of word embeddings and compositional functions, respectively. For the latter, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been considered. There has not been a rigorous evaluation regarding the relative importance of each component to different text-representation-based tasks; i.e., how important is the modeling capacity of word embeddings alone, relative to the added value of a compositional function? In this paper, we conduct an extensive comparative study between Simple Word Embeddings-based Models (SWEMs), with no compositional parameters, relative to employing word embeddings within RNN/CNN-based models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Moreover, in a new SWEM setup, we propose to employ a max-pooling operation over the learned word-embedding matrix of a given sentence. This approach is demonstrated to extract complementary features relative to the averaging operation standard to SWEMs, while endowing our model with better interpretability. To further validate our observations, we examine the information utilized by different models to make predictions, revealing interesting properties of word embeddings.
This paper empirically investigates the differences realized by using compositional functions over word embeddings as compared to directly operating the word embeddings. That is, the authors seek to explore the advantages afforded by RNN/CNN based models that induce intermediate semantic representations of texts, as opposed to simpler (parameter-free) approaches to composing these, like addition. 
In sum, I think this is exploration is interesting, and suggests that we should perhaps experiment more regularly with simple aggregation methods like SWEM. On the other hand, the differences across the models is relatively modest, and the data resists clear conclusions, so I'm not sure that the work will be very impactful. In my view, then, this work does constitute a contribution, albeit a modest one. I do think the general notion of attempting to simplify models until performance begins to degrade is a fruitful path to explore, as models continue to increase in complexity despite compelling evidence that this is always needed.
Strengths
---
+ This paper does highlight a gap in existing work, as far as I am aware: namely, I am not sure that there are generally known trade-offs associated with different compositional models over token embeddings for NLP. However, it is not clear that we should expect there to be a consistent result to this question across all NLP tasks.
+ The results are marginally surprising, insofar as I would have expected the CNN/RNN (particularly the former) to dominate the simpler aggregation approaches, and this does not seem borne out by the data. Although this trend is seemingly reversed on the short text data, muddying the story. 
Weaknesses
---
- There are a number of important limitations here, many of which the authors themselves note, which mitigate the implications of the reported results. First, this is a small set of tasks, and results may not hold more generally. It would have been nice to see some work on Seq2Seq tasks, or sequence tagging tasks at least. 
- I was surprised to see no mention of the "Fixed-Size Ordinally-Forgetting Encoding Method" (FOFE) proposed by Zhang et al. in 2015, which would seem to be a natural point of comparison here, given that it sits in a sweet spot of being simple and efficient while still expressive enough to preserve word-order information. This actually seems like a pretty glaring omission given that it meets many of the desiderata the authors put forward. 
- The interpretability angle discussed seems underdeveloped. I'm not sure that being able to identify individual words (as the authors have listed) meaningfully constitutes "interpretability" -- standard CNNs, e.g., lend themselves to this as well by tracing back through the filter activations. 
- Some of the questions addressed seem tangential to the main question of the paper -- e.g., word vector dimensionality seems an orthogonal issue to the composition function, and would influence performance for the more complex architectures as well.
Smaller comments
---
- On page 1, the authors write "By representing each word as a fixed-length vector, these embeddings can group semantically similar words, while explicitly encoding rich linguistic regularities and patterns", but actually I would say that these *implicitly* encode such regularities, rather than explicitly. 
- "architecture in Kim 2014; Collobert et al. 2011; Gan et al. 2017" -- citation formatting a bit weird here.
*** Update based on author response *** 
I have read the authors response and thank them for the additional details. 
Regarding the limited set of problems: of course any given work can only explore so many tasks, but for this to have general implications in NLP I would maintain that a standard (structured) sequence tagging task/dataset should have been considered. This is not about the number of datasets, but rather than diversity of the output spaces therein.
I appreciated the additional details regarding FOFE, which as the authors themselves note in their response is essentially a generalization of SWEM. 
Overall, the response has not changed my opinion on this paper: I think this (exploring simple representations and baselines) is an important direction in NLP, but feel that the paper would greatly benefit from additional work.