A Multiplicative Model for Learning Distributed Text-Based Attribute Representations
In this paper we propose a general framework for learning distributed representations of attributes: characteristics of text whose representations can be jointly learned with word embeddings. Attributes can correspond to a wide variety of concepts, such as document indicators (to learn sentence vectors), language indicators (to learn distributed language representations), meta-data and side information (such as the age, gender and industry of a blogger) or representations of authors. We describe a third-order model where word context and attribute vectors interact multiplicatively to predict the next word in a sequence. This leads to the notion of conditional word similarity: how meanings of words change when conditioned on different attributes. We perform several experimental tasks including sentiment classification, cross-lingual document classification, and blog authorship attribution. We also qualitatively evaluate conditional word neighbours and attribute-conditioned text generation.
This paper proposes to incorporate side information for improving vector-space embedding of words via an "attribute vector" that modulates the word-projection matrices. One could simply think of word-projection tensors (although, in practice the tensors are factorized) where the attribute vector provide the loadings for the tensor slices. This is studied in the context of log-bilinear language models, but the basic idea should be applicable to other word embedding work. The theory part of the paper is very well-written. However, it is in the experimental section that things get somewhat muddier. I would have preferred to see fewer experiments with greater depth than a barrage of experiments with little insight. Firstly, I am not sure what the experiments at the beginning of section 3 (contents of Tables 2 & 3) are really showing. Many models can generate text, and I am not sure these examples are showing anything in particular about the proposed model. Even a simple n-gram LM can interpolate between Bible and Caesar. Also, please mention what values of the initial learning rate, decay factor, momentum, and momentum increase factor were used. In section 3.1, what are the attributes? Is is just a sentence vector that is an average of all sub-phrases, or something else? Could you motivate why the choice is reasonable as an attribute? In section 3.2, please change the notation to make it clear that S is from l and S' and C_k are from l'. Perhaps v should really be v_l. Also, what are the attributes (x) in this case? In figure 1, right: you talk about a 1-hot attribute vector, but that's really the language-id vector and not the attribute vector. Also, do you have any insight as to why Germany and Deutschland are far apart while the translations of all other countries appear close together? Is it an error, some noise not accounted for, or showing something interesting about what the model learns (perhaps some us vs. them distinction, whereby Germany in English and Deutschland in German aren't learnt to be the same concept)?In section 3.3, if there are 234 unique attributes, why do you use a 100-dimensional attribute vector?