Convolutional Neural Network Architectures for Matching Natural Language Sentences
Semantic matching is of central importance to many natural language tasks [2,28]. A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them. As a step toward this goal, we propose convolutional neural network models for matching two sentences, by adapting the convolutional strategy in vision and speech. The proposed models not only nicely represent the hierarchical structures of sentences with their layerby-layer composition and pooling, but also capture the rich matching patterns at different levels. Our models are rather generic, requiring no prior knowledge on language, and can hence be applied to matching tasks of different nature and in different languages. The empirical study on a variety of matching tasks demonstrates the efficacy of the proposed model on a variety of matching tasks and its superiority to competitor models.
Summary:The paper presents two novel convolutional neural network architectures for modeling sentences in natural languages. These networks are trained specifically for the problem of matching a pair of sentences.The first architecture is a minor modification to the standard way of using a convolutional network over natural language sentences. After a convolution operation on the word embeddings, instead of doing a pooling operation across time (full sequence of words in a sentence) to select a single feature (or k features), the proposed model applies pooling to features associated with consecutive pairs of words.  This results in a output which is of half the size as the input. One can then apply multiple pairs of similar convolution-pooling layer to this output. In order to match pairs of sentences, this architecture is trained using a ranking loss applied to the final feature vector obtained from this network.The second architecture is far more novel than the first. It also includes convolutions over sentences, however instead of applying convolutions independently to each sentence, the model applies convolutions to all the possible pairs of windows on the pair of sentences to be matched. The idea is to enable the convolution kernels to capture the local interactions among words in the two sentences at low level, as opposed to at the level when the sentence representations have matured (as in the case of the first architecture).Both these models are evaluated on three tasks (two of them are somewhat artificial). They seem to beat the state-of-the-art (under certain conditions: discussed below).Quality:I think the models proposed in the paper are quite novel and certainly interesting (and useful) for others working in the same domain to try them out.However my biggest objection is with the evaluation of the proposed models.  The methodology of pre-training the word embeddings in an unsupervised way (word2vec in their case) and fixing them while training the subsequent convolutional neural network architecture is some what artificial and constrained. There is nothing stopping us from back propagating the gradients back into the word embeddings and tuning them to the task. Usually, when one does that, one observes a significant boost in performance. In particular, systems like senna (which are shown to be much worse than the proposed model in this paper) work extremely well when the embeddings are learnt for the task. My conjecture is that when the embeddings are allowed to learn for the task, architectures where pooling results in selection of a single (or k) features across the whole sentence are sufficient to capture both the internal structures of language objects and interactions among them.In summary, the experimental results as they stand are inconclusive. We do not yet know how superior the proposed model are against the state-of-the-art, and whether we are gaining anything significant.A few other points of note. * While the authors cite reference [9] (which uses k-max pooling in convolutional networks and have been shown to perform quite well), they do not compare it to their model. Any reason why? [9] recently has been shown to work quite well on standard nlp tasks. * The proposed model does not beat the state of the art in task 3 (Paraphrase Identification). It is understandable why is that the case: primarily lack of training data. However, interestingly enough, the authors do not show the numbers in Table 3.  Again, any reason why?Clarity:The paper is reasonably well written and easy to understand. Though it isriddled with typos which need to be fixed. An incomplete list of typos is: * 324: model,and > model, and * 325: the their best > the best * 334: hetergenous > heterogeneous * 373: requires less about > this whole sentence requires a complete re-phraseLastly, the details of their experimental setup were somewhat lacking as well. The values of the various model hyper-parameters used during the experiments are not clearly stated. Its also not clear whether the datasets used in experiments 1 and 2 are available for public use or not.Originality:The ideas proposed in the paper are fairly original and interesting.  Subject to their correct evaluation and validation, I think they can be very useful for others working on similar problem. The prior work is largely complete.Significance:I feel that the proposed models need to be properly evaluated before making any statements about their superiority and usefulness over previous techniques. However, assuming that they are validated correctly and shown to work well, then this work can potentially have a significant impact in NLP community where deep learning is only starting to show its usefulness.