Semi-supervised Convolutional Neural Networks for Text Categorization via Region Embedding
This paper presents a new semi-supervised framework with convolutional neural networks (CNNs) for text categorization. Unlike the previous approaches that rely on word embeddings, our method learns embeddings of small text regions from unlabeled data for integration into a supervised CNN. The proposed scheme for embedding learning is based on the idea of two-view semi-supervised learning, which is intended to be useful for the task of interest even though the training is done on unlabeled data. Our models achieve better results than previous approaches on sentiment classification and topic classification tasks.
like phrases, or regions of size 20 like sentences), eliminating the extra layer for word vector conversion. This direct learning of region embedding was noted to have the merit of higher accuracy with a simpler system (no need to tune hyper-parameters for word vectors) than supervised word vector-based CNN in which word vectors are randomly initialized and trained as part of CNN training. Moreover, the performance of [11]'s best CNN rivaled or exceeded the previous best results on the benchmark datasets.
Motivated by this finding, we seek effective use of unlabeled data for text categorization through direct learning of embeddings of text regions. Our new semi-supervised framework learns a region embedding from unlabeled data and uses it to produce additional input (additional to one-hot vectors) to supervised CNN, where a region embedding is trained with labeled data. Specifically, from unlabeled data, we learn tv-embeddings ('tv' stands for 'two-view'; defined later) of a text region through the task of predicting its surrounding context. According to our theoretical finding, a tv-embedding has desirable properties under ideal conditions on the relations between two views and the labels. While in reality the ideal conditions may not be perfectly met, we consider them as guidance in designing the tasks for tv-embedding learning.
We consider several types of tv-embedding learning task trained on unlabeled data; e.g., one task is to predict the presence of the concepts relevant to the intended task (e.g., 'desire to recommend the product') in the context, and we indirectly use labeled data to set up this task. Thus, we seek to learn tv-embeddings useful specifically for the task of interest. This is in contrast to the previous word vector/embedding learning methods, which typically produce a word embedding for general purposes so that all aspects (e.g., either syntactic or semantic) of words are captured. In a sense, the goal of our region embedding learning is to map text regions to high-level concepts relevant to the task. This cannot be done by word embedding learning since individual words in isolation are too primitive to correspond to high-level concepts. For example, "easy to use" conveys positive sentiment, but "use" in isolation does not. We show that our models with tv-embeddings outperform the previous best results on sentiment classification and topic classification. Moreover, a more direct comparison confirms that our region tv-embeddings provide more compact and effective representations of regions for the task of interest than what can be obtained by manipulation of a word embedding.
The paper presents an approach to learn features for region representation from unlabeled data, for use in text classification. The paper shows that using unlabeled data to estimate predictive features for text regions in a two-view learning framework brings substantial improvements over purely supervised learning of representations, and that learning region embeddings is substantially more effective than using word-embeddings (such as ones produced via word2vec).The paper explores different learning targets from unlabeled data as well as different representations of regions (bag-of-words vs concatenation). It is shown that the resulting representations are complementary and the combination is very effective.The main lesson from the paper: that learning region embeddings on unlabeled data is substantially more effective than the commonly employed method of using word-embeddings on unlabeled data and relying solely on them to represent regions, has substantial implications for NLP research.The paper is written very well and is easy to follow. The approach is original. The experiments are very thorough and bring insight into the performance of the different options.Improvements over the previous state-of-the-art obtained by use of deeper CNN models orsemi-supervised n-gram models are substantial and convincing.