Conic Scan-and-Cover algorithms for nonparametric topic modeling
We propose new algorithms for topic modeling when the number of topics is unknown. Our approach relies on an analysis of the concentration of mass and angular geometry of the topic simplex, a convex polytope constructed by taking the convex hull of vertices representing the latent topics. Our algorithms are shown in practice to have accuracy comparable to a Gibbs sampler in terms of topic estimation, which requires the number of topics be given. Moreover, they are one of the fastest among several state of the art parametric techniques.
1 Statistical consistency of our estimator is established under some conditions.
* Summary
This paper introduces a novel algorithm, Conic Scan Coverage, that is
based on convex geometry ideas and can perform non-parametric topic
modeling. The algorithm is intuitively based on the idea of covering
the topic simplex with cones. The papers presents the results of an
experimental evaluation and the supplementary contains detailed proofs
and example topics inferred by the algorithm.
* Evaluation
I have very mixed feelings about this paper.
On the one hand, I must say I had a lot of fun reading it. I am really
fond of the convex geometric approach to topic modeling, it is such an
original and potentially promising approach. Indeed, the results are
very good, and the topics presented in the supplementary material look
great.
On the other hand, I have a lot of uncertainty about this
paper. First, I must confess I do not understand the paper as well as
I would like. Of course, it could mean that I should have tried
harder, or maybe that the authors should explain more. Probably a bit
of both. Second, the experiments are weak. However, the paper is so
original that I don't think it should be rejected just because of
this.
To summarize:
+ Impressive results: excellent runtime and statistical performance, good looking topics
- Weak experimental evaluation
+ Very original approach to topic modeling
- Algorithm 1 seems very ad-hoc, and justification seems insufficient to me
Could you please show me a few example of inferred topic distribution?
* Discussion
- The experimental evaluation could be much better.
  The python implementation of Gibbs-LDA is pretty bad, it makes for a
  very poor baseline. You really need to say somewhere how many
  iterations were run.
  When comparing against algorithms like LDA Gibbs or HDP Gibbs, as in
  table 1, you can't report a single number. You should plot the
  evolution of perplexity by iteration. For large datasets, Gibbs LDA
  can reach a good perplexity and coherence in surprisingly few
  iterations. Also, there should be some error bars, a number in
  itself is not very convincing.
  You should also share information about hyperparameter settings. For
  instance, HDP and LDA can exhibit very different perplexity for
  different values of their alpha and beta hyperparameters.
  Finally, you should also vary the vocabulary size. I did note that V
  is very small, and it is worrysome. There are been several
  algorithms proposed in the past that seemed very effective for
  unrealistic small vocabulary sizes and didn't scale well to this
  parameter.
- The algorithm seems very ad-hoc
  I am surprised that you choose a topic as the furthest document and
  then remove all the documents within some cosine distance. Why makes
  such a farthest document a good topic representative? Also, if we
  remove the documents from the active sets based solely on one topic,
  are we really trying to explain the documents as mixture of topics?
  I would be really curious to see a few documents and their inferred
  topic distributions to see if it is interpretable.
- What good is Theorem 4?
  Theorem 4 is very interesting and definitely provides some good
  justification for the algorithm. However, it assumes that the number
  of topics K is known and doesn't say much about why such a procedure
  should find a good number of topics. Indeed, in the end, I don't
  have much intuition about what exactly drives the choice of number
  of topics.