A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION
To represent a text as a bag of properly identified "phrases" and use the representation for processing the text is proved to be useful. The key question here is how to identify the phrases and represent them. The traditional method of utilizing n-grams can be regarded as an approximation of the approach. Such a method can suffer from data sparsity, however, particularly when the length of n-gram is large. In this paper, we propose a new method of learning and utilizing task-specific distributed representations of n-grams, referred to as "region embeddings". Without loss of generality we address text classification. We specifically propose two models for region embeddings. In our models, the representation of a word has two parts, the embedding of the word itself, and a weighting matrix to interact with the local context, referred to as local context unit. The region embeddings are learned and used in the classification task, as parameters of the neural network classifier. Experimental results show that our proposed method outperforms existing methods in text classification on several benchmark datasets. The results also indicate that our method can indeed capture the salient phrasal expressions in the texts.
The authors present a model for text classification. The parameters of the model are an embedding for each word and a local context unit. The local context unit can be seen as a filter for a convolutional layer, but which filter is used at location i depends on the word at location i (i.e. there is one filter per vocabulary word). After the filter is applied to the embeddings and after max pooling, the word-context region embeddings are summed and fed into a neural network for the classification task. The embeddings, the context units and the neural net parameters are trained jointly on a supervised text classification task. The authors also offer an alternative model, which changes the role of the embedding an the context unit, and results in context-word region embeddings. Here the embedding of word i is combined with the elements of the context units of words in the context. To get the region embeddings both model (word-context and context-word) combine attributes of the words (embeddings) with how their attributes should be emphasized or deemphasized based on nearby words (local context units and max pooling) while taking into account the relative position of the words in the context (columns of the context units). 
The method beats existing methods for text classification including d-LSTMs , BoWs, and ngram TFIDFs on held out classification accuracy. the choice of baselines is convincing. What is the performance of the proposed method if the embeddings are initialized to pretrained word embeddings and a) trained for the classification task together with randomly initialized context units b) frozen to pretrained embeddings and only the context units are trained for the classification task?
The introduction was fine. Until page 3 the authors refer to the context units a couple of times without giving some simple explanation of what it could be. A simple explanation in the introduction would improve the writing.
The related work section only makes sense *after* there is at least a minimal explanation of what the local context units do. A simple explanation of the method, for example in the introduction, would then make the connections to CNNs more clear. Also, in the related work, the authors could include more citations (e.g. the d-LSTM and the CNN based methods from Table 2) and explain the qualitative differences between their method and existing ones.
The authors should consider adding equation numbers. The equation on the bottom of page 3 is fine, but the expressions in 3.2 and 3.3 are weird. A more concise explanation of the context-word region embeddings and the word-context region embeddings would be to instead give the equation for r_{i,c}.  
The included baselines are extensive and the proposed method outperforms existing methods on most datasets. In section 4.5 the authors analyze region and embedding size, which are good analyses to include in the paper. Figure 2 and 3 could be next to each other to save space. 
I found the idea of multi region sizes interesting, but no description is given on how exactly they are combined. Since it works so well, maybe it could be promoted into the method section? Also, for each data set, which region size worked best?
Qualitative analysis: It would have been nice to see some analysis of whether the learned embeddings capture semantic similarities, both at the embedding level and at the region level. It would also be interesting to investigate the columns of the context units, with different columns somehow capturing the importance of relative position. Are there some words for which all columns are similar meaning that their position is less relevant in how they affect nearby words? And then for other words with variation along the columns of the context units, do their context units modulate the embedding more when they are closer or further away? 
Pros:
 + simple model
 + strong quantitative results
Cons:
 - notation (i.e. precise definition of r_{i,c})
 - qualitative analysis could be extended
 - writing could be improved