which neural embedding model works better?
the CRX model
Explicit concept space models have proven efficacy for text representation in many natural language and text mining applications. The idea is to embed textual structures into a semantic space of concepts which captures the main topics of these structures. That so called bag-of-concepts representation suffers from data sparsity causing low similarity scores between similar texts due to low concept overlap. In this paper we propose two neural embedding models in order to learn continuous concept vectors. Once learned, we propose an efficient vector aggregation method to generate fully dense bag-of-concepts representations. Empirical results on a benchmark dataset for measuring entity semantic relatedness show superior performance over other concept embedding models. In addition, by utilizing our efficient aggregation method, we demonstrate the effectiveness of the densified vector representation over the typical sparse representations for dataless classification where we can achieve at least same or better accuracy with much less dimensions.
Vector-based semantic mapping models are used to represent textual structures (words, phrases, and documents) as high-dimensional meaning vectors. Typically, these models utilize textual corpora and/or Knowledge Bases (KBs) to acquire world knowledge, which is then used to generate a vector representation for the given text in the semantic space. The goal is thus to accurately place semantically similar structures close to each other in that semantic space. On the other hand, dissimilar structures should be far apart. Explicit concept space models are motivated by the idea that high level cognitive tasks such learning and reasoning are supported by the knowledge we acquire from concepts BIBREF0 . Therefore, such models utilize concept vectors (a.k.a bag-of-concepts (BOC)) as the underlying semantic representation of a given text through a process called conceptualization, which is mapping the text into relevant concepts capturing its main topics. The concept space typically include concepts obtained from KBs such as Wikipedia, Probase BIBREF1 , and others. Once the concept vectors are generated, similarity between two concept vectors can be computed using a suitable similarity/distance measure such as cosine. The BOC representation has proven efficacy for semantic analysis of textual data especially short texts where contextual information is missing or insufficient. For example, measuring semantic similarity/relatedness BIBREF2 , BIBREF3 , BIBREF4 , dataless classification BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , short text clustering BIBREF0 , search and relevancy ranking BIBREF9 , event detection and coreference resolution BIBREF10 . Similar to the traditional bag-of-words representation, the BOC vector is a high dimensional sparse vector whose dimensionality is the same as the number of concepts in the employed KB (typically millions). Consequently, it suffers from data sparsity causing low similarity scores between similar texts due to low concept overlap. Formally, given a text snippet INLINEFORM0 of INLINEFORM1 terms where INLINEFORM2 , and a concept space INLINEFORM3 of size INLINEFORM4 . The BOC vector INLINEFORM5 = INLINEFORM6 of INLINEFORM7 is a vector of weights of each concept where each INLINEFORM8 of concept INLINEFORM9 is calculated as in equation 1: DISPLAYFORM0  Here INLINEFORM0 is a scoring function which indicates the degree of association between term INLINEFORM1 and concept INLINEFORM2 . For example, gabrilovich2007computing proposed Explicit Semantic Analysis (ESA) which uses Wikipedia articles as concepts and the TF-IDF score of the terms in these article as the association score. Another scoring function might be the co-occurrence count or Pearson correlation score between INLINEFORM3 and INLINEFORM4 . As we can notice, only very small subset of the concept space would have non-zero scores with the given terms. Moreover, the BOC vector is generated from the INLINEFORM5 concepts which have relatively high association scores with the input terms (typically few hundreds). Thus each text snippet is mapped to a very sparse vector of millions of dimensions having only few hundreds non-zero values BIBREF10 . Typically, the cosine similarity measure is used compute the similarity between a pair of BOC vectors INLINEFORM0 and INLINEFORM1 . Because the concept vectors are very sparse, we can rewrite each vector as a vector of tuples INLINEFORM2 . Suppose that INLINEFORM3 and INLINEFORM4 , where INLINEFORM5 and INLINEFORM6 are the corresponding weights of concepts INLINEFORM7 and INLINEFORM8 respectively. And INLINEFORM9 , INLINEFORM10 are the indices of these concepts in the concept space INLINEFORM11 such that INLINEFORM12 . Then, the relatedness score can be written as in equation 2: DISPLAYFORM0  where INLINEFORM0 is the indicator function which returns 1 if INLINEFORM1 and 0 otherwise. Having such sparse representation and using exact match similarity scoring measure, we can expect that two very similar text snippets might have zero similarity score if they map to different but very related set of concepts BIBREF7 . Neural embedding models have been proposed to overcome the BOC sparsity problem. The basic idea is to learn fixed size continuous vectors for each concept. These vectors can then be used to compute concept-concept similarity and thus overcome the concept mismatch problem. In this paper we propose two neural embedding models in order to learn continuous concept vectors based on the skip-gram model BIBREF11 . Our first model is the Concept Raw Context model (CRC) which utilizes concept mentions in a large scale KB to jointly learn embeddings of both words and concepts. Our second model is the Concept-Concept Context model (3C) which learns the embeddings of concepts from their conceptual contexts (i.e., contexts containing surrounding concepts only). After learning the concept vectors, we propose an efficient concept vector aggregation method to generate fully dense BOC representations. Our efficient aggregation method allows measuring the similarity between pairs of BOC vectors in linear time. This is more efficient than prior methods which require quadratic time or at least log-linear time if optimized (see equation 2). We evaluate our embedding models on two tasks: The contributions of this paper are threefold: First, we propose two low cost concept embedding models which requires few hours rather than days to train. Second, we propose simple and efficient vector aggregation method to obtain fully densified BOC vectors in linear time. Third, we demonstrate through experiments that we can obtain same or better accuracy using the densified BOC representation with much less dimensions (few in most cases), reducing the computational cost of generating the BOC vector significantly.