textTOvec: DEEP CONTEXTUALIZED NEURAL AUTOREGRESSIVE TOPIC MODELS OF LANGUAGE WITH DISTRIBUTED COMPOSITIONAL PRIOR
We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i.e., P (word|context) : (1) No language structure in context: Probabilistic topic models ignore word order by summarizing a given context as a "bag-of-word" and consequently the semantics of words in the context is lost. In this work, we incorporate language structure by combining a neural autoregressive topic model (TM) (e.g., DocNADE) with a LSTM based language model (LSTM-LM) in a single probabilistic framework. The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns, while the TM simultaneously learns a latent representation from the entire document. In addition, the LSTM-LM models complex characteristics of language (e.g., syntax and semantics), while the TM discovers the underlying thematic structure in a collection of documents. We unite two complementary paradigms of learning the meaning of word occurrences by combining a topic model and a language model in a unified probabilistic framework, named as ctx-DocNADE. (2) Limited context and/or smaller training corpus of documents: In settings with a small number of word occurrences (i.e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging. We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus. The proposed Doc-NADE extension is named as ctx-DocNADEe. We present novel neural autoregressive topic model variants coupled with neural language models and embeddings priors that consistently outperform state-of-theart generative topic models in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classification) over 7 long-text and 8 short-text datasets from diverse domains.
The paper extends an existing topic model - DocNADE - by replacing the feedforward part of the network which combines the textual context with an LSTM sequence model. Hence this paper fits in a long tradition of work which aims to extend the bad-of-words model from the original LDA paper with some sequence information.
The authors do a commendable job in thoroughly evaluating the proposed extension, using a number of evaluations based on perplexity, topic coherence, and text retrieval and categorization.
My main problem with the paper as it stands is that it a) arguably oversells the contribution, and b) is unclear when explaining certain crucial aspects of the model.
It would also help to have a clearer statement of whether the contribution here is on the document modeling side, or the language modeling side. Motivation is provided from both angles, but the evaluation focuses largely on the topic modeling (which is fine, just need to say it).
More specific comments:
--
The abstract should mention that the DocNADE model already exists, and that the contribution of the current work is to extend that existing model in a particular way. For those readers unfamiliar with DocNADE, this will help situate the work with regard to the existing literature.
Using existing word embeddings as a 'prior' for the LSTM word embeddings is a completely standard alternative now to learning those embeddings from scratch. I'm not sure that can count as a second, major contribution of the paper. (I'm also not sure that either extension to DocNADE warrants a new name, but I'll leave that to the authors' judgement.)
I'm confused by one aspect of the DocNADE model: "the topic assigned ...equally depends all the other words appearing in the same document". But the model is generative, no? And eqn 1 suggests that each word is generated conditioned on the *previous* words in the document, or did I miss something? 
Related to this point, DocNADE transforms its BoWs into a sequence. But what's the order? Is it just the order of the words in the document? In which case it's very similar to the LSTM extension, except the LSTM keeps the order in the history, whereas the bag-of-words model doesn't.
Relation to generative models: LDA is a generative model with a generative story. It's not completely obvious to me what the generative story is in the new model. Talking about "distributed compositional priors" doesn't help, since I'm assuming these aren't priors in a Bayesian modeling sense? (It's also not clear in what sense these "priors" are compositional, but that's a separate question.)
Equation 2: what's the motivation for mixing the LSTM history with the bag-of-words (esp. if the history is from the same bag of words in each case). Why not just use the LSTM?
It would be useful to state in the main body of the text what the value of lambda ends up being. In 3.1 there's a suggestion this might be 0.01, but that effectively ignores the LSTM?
Similar question: how can the DocNADE model provide a *global* context if the model is generative?
Perplexity is a reasonable thing to measure, but presumably the auto-regressive nature of the LSTM means that it's more-or-less guaranteed to do better than a bag-of-words model? I wonder if it's worth acknowledging this fact?
I don't understand why lambda has to be zero "to compute the exact log-likelihood".
The first line of the conclusion doesn't say much: it's pretty obvious that the ordering of the words is going to help better estimate the probability of a word in a given context; 50 years of language modeling research has already taught us that.
Minor presentational comments:
--
Some of the hyphenation looks odd, eg in the title. Are you using the standard LaTeX hyphenation settings?
Strictly speaking, I'm not sure that 'bear' in the example is a proper noun.
"orderless sets of words": bags, not sets, since the counts matter, no?
The tables are too small, with a lot of numbers in them. One option is to move some of the details to the Appendix. Either way there needs to be more summary in the main body explaining what the numbers tell us.