what datasets where used?
STSB, SICK, MRPC
Different word embedding models capture different aspects of linguistic properties. This inspired us to propose a model (M-MaxLSTM-CNN) for employing multiple sets of word embeddings for evaluating sentence similarity/relation. Representing each word by multiple word embeddings, the MaxLSTM-CNN encoder generates a novel sentence embedding. We then learn the similarity/relation between our sentence embeddings via Multi-level comparison. Our method M-MaxLSTM-CNN consistently shows strong performances in several tasks (i.e., measure textual similarity, identify paraphrase, recognize textual entailment). According to the experimental results on STS Benchmark dataset and SICK dataset from SemEval, M-MaxLSTM-CNN outperforms the state-of-the-art methods for textual similarity tasks. Our model does not use hand-crafted features (e.g., alignment features, Ngram overlaps, dependency features) as well as does not require pre-trained word embeddings to have the same dimension.
Measuring the semantic similarity/relation of two pieces of short text plays a fundamental role in a variety of language processing tasks (i.e., plagiarism detection, question answering, and machine translation). Semantic textual similarity (STS) task is challenging because of the diversity of linguistic expression. For example, two sentences with different lexicons could have a similar meaning. Moreover, the task requires to measure similarity at several levels (e.g., word level, phrase level, sentence level). These challenges give difficulties to conventional approaches using hand-crafted features. Recently, the emergence of word embedding techniques, which encode the semantic properties of a word into a low dimension vector, leads to the successes of many learning models in natural language processing (NLP). For example, BIBREF0 randomly initialize word vectors, then tunes them during the training phase of a sentence classification task. By contrast, BIBREF1 initialize word vectors via the pre-train word2vec model trained on Google News BIBREF2 . BIBREF3 train a word embedding model on the paraphrase dataset PPDB, then apply the word representation for word and bi-gram similarity tasks. Several pre-trained word embeddings are available, which are trained on various corpora under different models. BIBREF4 observed that different word embedding models capture different aspects of linguistic properties: a Bag-of-Words contexts based model tends to reflect the domain aspect (e.g., scientist and research) while a paraphrase-relationship based model captures semantic similarities of words (e.g., boy and kid). From experiments, we also observed that the performance of a word embedding model is usually inconsistent over different datasets. This inspired us to develop a model taking advantages of various pre-trained word embeddings for measuring textual similarity/relation. In this paper, we propose a convolutional neural network (CNN) to learn a multi-aspect word embedding from various pre-trained word embeddings. We then apply the max-pooling scheme and Long Short Term Memory (LSTM) on this embedding to form a sentence representation. In STS tasks, BIBREF5 shows the efficiency of the max-pooling scheme in modeling sentences from word embedding representations refined via CNN. However, the max-pooling scheme lacks the property of word order (e.g., sentence(“Bob likes Marry”) = sentence(“Marry likes Bob”)). To address this weakness, we use LSTM as an additional scheme for modeling sentences with word order characteristics. For measuring the similarity/relation between two sentence representations, we propose Multi-level comparison which consists of word-word level, sentence-sentence level, and word-sentence level. Through these levels, our model comprehensively evaluates the similarity/relation between two sentences. We evaluate our M-MaxLSTM-CNN model on three tasks: STS, textual entailment recognition, paraphrase identification. The advantages of M-MaxLSTM-CNN are: i) simple but efficient for combining various pre-trained word embeddings with different dimensions; ii) using Multi-level comparison shows better performances compared to using only sentence-sentence comparison; iii) does not require hand-crafted features (e.g., alignment features, Ngram overlaps, syntactic features, dependency features) compared to the state-of-the-art ECNU BIBREF6 on STS Benchmark dataset. Our main contributions are as follows: The remainder of this paper is organized as follows: Section 2 reviews the previous research, Section 3 introduces the architecture of our model, Section 4 describes the three tasks and datasets, Section 5 describes the experiment setting, Section 6 reports and discusses the results of the experiments, and Section 7 concludes our work.