  Neural topic models (NTMs) apply deep neural networks to topic modelling.
Despite their success, NTMs generally ignore two important aspects: (1) only
document-level word count information is utilized for the training, while more
fine-grained sentence-level information is ignored, and (2) external semantic
knowledge regarding documents, sentences and words are not exploited for the
training. To address these issues, we propose a variational autoencoder (VAE)
NTM model that jointly reconstructs the sentence and document word counts using
combinations of bag-of-words (BoW) topical embeddings and pre-trained semantic
embeddings. The pre-trained embeddings are first transformed into a common
latent topical space to align their semantics with the BoW embeddings. Our
model also features hierarchical KL divergence to leverage embeddings of each
document to regularize those of their sentences, thereby paying more attention
to semantically relevant sentences. Both quantitative and qualitative
experiments have shown the efficacy of our model in 1) lowering the
reconstruction errors at both the sentence and document levels, and 2)
discovering more coherent topics from real-world datasets.

Computation and Language;Machine Learning