On the Use of Word Embeddings Alone to Represent Natural Language Sequences
To construct representations for natural language sequences, information from two main sources needs to be captured: (i) semantic meaning of individual words, and (ii) their compositionality. These two types of information are usually represented in the form of word embeddings and compositional functions, respectively. For the latter, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been considered. There has not been a rigorous evaluation regarding the relative importance of each component to different text-representation-based tasks; i.e., how important is the modeling capacity of word embeddings alone, relative to the added value of a compositional function? In this paper, we conduct an extensive comparative study between Simple Word Embeddings-based Models (SWEMs), with no compositional parameters, relative to employing word embeddings within RNN/CNN-based models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Moreover, in a new SWEM setup, we propose to employ a max-pooling operation over the learned word-embedding matrix of a given sentence. This approach is demonstrated to extract complementary features relative to the averaging operation standard to SWEMs, while endowing our model with better interpretability. To further validate our observations, we examine the information utilized by different models to make predictions, revealing interesting properties of word embeddings.
 This paper extensively compares simple word embedding based models (SWEMs) to RNN/CNN based-models on a suite of NLP tasks. Experiments on document classification, sentence classification, and natural language sequence matching show that SWEMs perform competitively or even better in the majority of cases. The authors also propose to use max pooling to complement average pooling for combining information from word embeddings in a SWEM model to improve interpretability. While there is not much contribution in terms of technical novelty, I think this is an interesting paper that sheds new lights on limitations of existing methods for learning sentence and document representations. The paper is well written and the experiments are quite convincing. - An interesting finding is that word embeddings are better for longer documents, whereas RNN/CNN models are better for shorter text. Do the authors have any sense on whether this is because of the difficulty in training an RNN/CNN model for long documents or whether compositions are not necessary since there are multiple predictive independent cues in a long text? - It would be useful to include a linear classification model that takes the word embeddings as an input in the comparison (SWEM-learned). - How crucial is it to retrain the word embeddings on the task of interest (from GloVe initialization) to obtain good performance?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
On the Use of Word Embeddings Alone to Represent Natural Language Sequences
To construct representations for natural language sequences, information from two main sources needs to be captured: (i) semantic meaning of individual words, and (ii) their compositionality. These two types of information are usually represented in the form of word embeddings and compositional functions, respectively. For the latter, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have been considered. There has not been a rigorous evaluation regarding the relative importance of each component to different text-representation-based tasks; i.e., how important is the modeling capacity of word embeddings alone, relative to the added value of a compositional function? In this paper, we conduct an extensive comparative study between Simple Word Embeddings-based Models (SWEMs), with no compositional parameters, relative to employing word embeddings within RNN/CNN-based models. Surprisingly, SWEMs exhibit comparable or even superior performance in the majority of cases considered. Moreover, in a new SWEM setup, we propose to employ a max-pooling operation over the learned word-embedding matrix of a given sentence. This approach is demonstrated to extract complementary features relative to the averaging operation standard to SWEMs, while endowing our model with better interpretability. To further validate our observations, we examine the information utilized by different models to make predictions, revealing interesting properties of word embeddings.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper extensively compares simple word embedding based models (SWEMs) to RNN/CNN based-models on a suite of NLP tasks. Experiments on document classification, sentence classification, and natural language sequence matching show that SWEMs perform competitively or even better in the majority of cases. The authors also propose to use max pooling to complement average pooling for combining information from word embeddings in a SWEM model to improve interpretability. While there is not much contribution in terms of technical novelty, I think this is an interesting paper that sheds new lights on limitations of existing methods for learning sentence and document representations. The paper is well written and the experiments are quite convincing. - An interesting finding is that word embeddings are better for longer documents, whereas RNN/CNN models are better for shorter text. Do the authors have any sense on whether this is because of the difficulty in training an RNN/CNN model for long documents or whether compositions are not necessary since there are multiple predictive independent cues in a long text? - It would be useful to include a linear classification model that takes the word embeddings as an input in the comparison (SWEM-learned). - How crucial is it to retrain the word embeddings on the task of interest (from GloVe initialization) to obtain good performance?
assistant
 This paper extensively compares simple word embedding based models (SWEMs) to RNN/CNN based-models on a suite of NLP tasks. Experiments on document classification, sentence classification, and natural language sequence matching show that SWEMs perform competitively or even better in the majority of cases. The authors also propose to use max pooling to complement average pooling for combining information from word embeddings in a SWEM model to improve interpretability. While there is not much contribution in terms of technical novelty, I think this is an interesting paper that sheds new lights on limitations of existing methods for learning sentence and document representations. The paper is well written and the experiments are quite convincing. - An interesting finding is that word embeddings are better for longer documents, whereas RNN/CNN models are better for shorter text. Do the authors have any sense on whether this is because of the difficulty in training an RNN/CNN model for long documents or whether compositions are not necessary since there are multiple predictive independent cues in a long text? - It would be useful to include a linear classification model that takes the word embeddings as an input in the comparison (SWEM-learned). - How crucial is it to retrain the word embeddings on the task of interest (from GloVe initialization) to obtain good performance?