In this paper we present a model for unsupervised topic discovery in texts corpora. The proposed model uses documents, words, and topics lookup table embedding as neural network model parameters to build probabilities of words given topics, and probabilities of topics given documents. These probabilities are used to recover by marginalization probabilities of words given documents. For very large corpora where the number of documents can be in the order of billions, using a neural auto-encoder based document embedding is more scalable then using a lookup table embedding as classically done. We thus extended the lookup based document embedding model to continuous auto-encoder based model. Our models are trained using probabilistic latent semantic analysis (PLSA) assumptions. We evaluated our models on six datasets with a rich variety of contents. Conducted experiments demonstrate that the proposed neural topic models are very effective in capturing relevant topics. Furthermore, considering perplexity metric, conducted evaluation benchmarks show that our topic models outperform latent Dirichlet allocation (LDA) model which is classically used to address topic discovery tasks.
This paper proposes a neural topic model that aim to discover topics by minimizing a version of the PLSA loss. According to PLSA, a document is presented as a mixture of topics, while a topic is a probability distribution over words, with documents and words assumed independent given topics. Thanks to this assumption, each of these probability distributions (word|topic, topic|document, and word|document) can essentially be expressed as a matrix multiplication of the other two, and EM is usually adopted for the optimization. This paper proposes to embed these relationships in a neural network and then optimize the model using SGD.
I believe the paper should be rejected because: 1) most aspects of this paper are a little dated 2) novelty is little 3) experimental section is very limited and unconvincing.
To elaborate on the experimental section:
- Only LDA has been presented as baseline. There's plenty of neural topic models to compare against (you mentioned some in your related work section) but no comparison with any of those is presented. If the concern is their training time on large datasets, they should be at least presented as comparison for the smaller datasets. For the large datasets there's other approaches that would scale and should be presented as baselines: 1) train on a sample of the dataset 2) co-occurrence based topic methods on sliding windows of text are extremely fast (eg see "A Biterm Topic Model", "A Practical Algorithm for Topic Modeling with Provable Guarantees", and "A Reduction for Efficient LDA Topic Reconstruction" which could fit your scenario with large datasets where topics most likely have small overlap with each other and are almost separable by anchor words.)
- Even regarding just LDA: what hyper-parameters \alpha and \beta did you set for LDA? Tuning \beta to a small value might have an impact for large datasets.
- Metrics: only perplexity is presented and metrics but it's well known that perplexity on its own is quite limited and often is not correlated to human judgment. Consider adding topic coherence measures as well.
- The section on continuous document embeddings is confusing and the explanation should be improved and the formalism tightened.
Other (did not impact the score):
- Biases: you're adding biases to your probability estimation equations. This is not in line  with the PLSA assumption. What happens if no biases are used?
The paper has several typos and grammatical errors, e.g.:
- page 2, L#1: networks -> network
- page 4, sec 3.2: set unobserved -> set of unobserved
- page 5, sec 5: pratise -> practice
- several places: it's -> its