Analysis of Variational Bayesian Latent Dirichlet Allocation: Weaker Sparsity than MAP
Latent Dirichlet allocation (LDA) is a popular generative model of various objects such as texts and images, where an object is expressed as a mixture of latent topics. In this paper, we theoretically investigate variational Bayesian (VB) learning in LDA. More specifically, we analytically derive the leading term of the VB free energy under an asymptotic setup, and show that there exist transition thresholds in Dirichlet hyperparameters around which the sparsity-inducing behavior drastically changes. Then we further theoretically reveal the notable phenomenon that VB tends to induce weaker sparsity than MAP in the LDA model, which is opposed to other models. We experimentally demonstrate the practical validity of our asymptotic theory on real-world Last.FM music data.
In this paper, authors analyze sparsity of the posterior parameters in LDA using a variational Bayesian algorithm. They derive an expression for the VB free energy which shows its asymptotic behaviour with respect to number of words (N), number of documents (M), vocabulary size (L) etc. Their results suggest that, for certain settings of L,M,N,  the sparsity behaviour changes drastically at a particular hyper-parameter setting. These changes differ from those of MAP and partial-Bayes algorithms.The problem discussed in this paper is original, interesting, and is perhaps useful too.  The arguments are supported by rigorous theorems. I found the paper to be really hard to read and understand. The paper is not written clearly, and there are many steps in the derivations that are not outlined. Therefore, I was unable to understand the derivation the theorem. I have few suggestions below to improve this part. Another big problem that I find with this work is the context. I think authors need to discuss the significance of their results in the context of real data. Perhaps, a more extensive experiment section will be useful. I have some suggestions on this below.Detailed comments:The paper discusses, at several occasion (e.g. second paragraph), sparsity in general latent variable models (e.g. mixture model etc.). If the notion of sparsity is comparable in these models, then it will be good to include a general definition of sparsity in latent variable models. Otherwise, it will be good to include a discussion of what exactly is meant by weaker or stronger sparsity in these models, e.g. mixture model, and how does it compare to sparsity in LDA.The LDA notation used in the paper is not-standard and makes it difficult to read for people familiar with LDA. I highly recommend to change the notation. Also, please include a graphical model in Section 2.1. Use q for approximate distribution instead of r. Paragraph at line 185 is not clear. It is perhaps clearer to state that you assume A* and B* of rank K* that can model the data well.Define O_p for people who are not familiar with it.Explain more clearly the average number of nonzero entries in line 201.In Line 253, you say "Note that Theorem 1 combined with Lemma 1 implies the consistency". This isn't trivial (at least to me), although I can see how to get there. It is perhaps a good idea to write these results as theorem rather than the Lemma that you have. At least it gives a clear statement to the reader about what is being proved in the paper. You should then include a detailed proof (really detailed) in the Appendix. I will suggest the same for Corollary 1 and 2. I do not see how you arrive at these results so trivially. It will be good have details on this in the Appendix. The proofs in Appendix need to be more detailed too. Currently, there are statements that are not easy to derive, or at least I could not derive them myself. E.g. line 511, "because of the independence between A and B, the posterior distribution must converge to a point... ". I don't know why? In line 522, "we can show that...". Unfortunately, I wasn't able to show this, so some help will make the experience for readers less frustrating. It was so discouraging that I couldn't make it beyond the first page of Appendix.Please provide some context for Corollary 1 and 2. You have discussed two settings of L,M,N where you have clear statements. What are the significance of these cases? when they are likely to occur? do real data have one setting more than others? If we only know about sparsity of B, can we still conclude something general about the whole situation?Please make a table comparing PBB, PBA, MAP and VB vs. settings of c_a and c_b. This will improve clarity in summarizing the results. It seems right now that integrating A out gives the sparsity at c_a=1/2 behaviour. Do you think that will be true in general?Line 305, MAP tends to induce stronger sparsity. Please clarify this further. I assume you mean that VB does not include c_a >1/2 cases therefore will have less sparsity? Line 323. L and M would be comparable to N in standard text analysis. I don't think it's correct to say that.Results section. Why don't you include other text datasets on which LDA has been applied more frequently rather than LastFM? Also provide a reference that applies LDA on lastFM.Please include more datasets. Remove c_b from plots if it doesn't affect the sparsity. Rather include more datasets and more different scenarios for different values of L,M, and N.