Finite-Dimensional BFRY Priors and Variational Bayesian Inference for Power Law Models
Bayesian nonparametric methods based on the Dirichlet Process (DP), gamma process and beta process, have proven effective in capturing aspects of various datasets arising in machine learning. However, it is now recognized that such processes have their limitations in terms of the ability to capture power law behavior. As such there is now considerable interest in models based on the Stable Processs (SP), Generalized Gamma process (GGP) and Stable-Beta Process (SBP). These models present new challenges in terms of practical statistical implementation. In analogy to tractable processes such as the finite-dimensional Dirichlet process, we describe a class of random processes, we call iid finite-dimensional BFRY processes, that enables one to begin to develop efficient posterior inference algorithms such as variational Bayes that readily scale to massive datasets. For illustrative purposes, we describe a simple variational Bayes algorithm for normalized SP mixture models, and demonstrate its usefulness with experiments on synthetic and real-world datasets.
Priors for random measures that induces power-law distributions (such as the GGP) has recently captured a great deal of attention in Bayesian non-parametrics. Inference in models based on these measures is often made difficult because of their analytical properties, for instance the difficulty in obtaining a simple representation of the posterior distribution. This makes existing samplers slow and difficult to implement and makes the use of other inference techniques such as VB difficult. The authors propose a finite-dimensional (truncated) representation of the underlying random measure based on the BFRY distribution which allows simpler sampling procedures and easy construction of VB inference. The construction is tested on realistic data sets.
This paper addresses a timely and important problem in modern non-parametrics, efficient inference for an important class of power-law inducing random measures. I was not aware of the BFRY distribution and pointing out tools such as this, and how they can be implemented, is in my view a good contributions to this conference. I must however admit I did not understand all details of the paper and I hope other reviewers will be able to confirm it's technical accuracy.   I understand this is a difficult and technical topic however I think the authors should improve the presentation. Many difficult topics are introduced or mentioned very quickly and it would serve the paper well if any of these could be omitted for brevity. Is it necessary to introduce the sigma-stable random variable, Kingmann processes, and tilting in the *introduction*? A suggestion could be to introduce the representation eq. (14) in the introduction as an equation (perhaps with generic distributions instead of the BFRY) to provide the reader with a clear picture of what we want to end up with and what the concept of "ideal" finite-dimensional distribution refers to. If the authors need more space, eq.13-15 contains some redundancy.  In regards to the simulations, I think the graphs needs to be cleaned up. In my printed version they are hardly readable and more thought should be given to their readability (font/line sizes, data area, etc.). Should the figure on p.4 not be named? I would also recommend giving the references size/spacing some more thought (are you sure this agrees with the NIPS format?).   I would suggest not showing the difference in figure 1 but the absolute value. Later, the error bars overlap in a way that makes the graphic very difficult to read. I think the tables should be cleaned up to either exclude variance or show it in bolded; quite frankly I got lost in terms of what conclusions I should draw from the tables because of the many abbreviations and it would be recommendable to show a subset of the  results and better communicate what the conclusions are. I would also suggest showing a single number and bolding “best” entries.   A thing I missed a lot from the simulations are results showing that the power-law behavior of the finite-dimensional representation is easily preserved. Playing the devils advocate, the prior distribution (DP or something with a power low behavior) is sometimes not that important in data modelling, and therefore a good test log-likelihood in the actual models does not necessarily show the finite-dimensional representation preserves power-law behavior.  It would in my view benefit the paper if it could be shown by simulation for which K (compared to some dataset size simulated from a power-law distribution such as a GGP) power-law behavior in the posterior is preserved by the finite-dimensional representation. One could expect the tail distribution to require a lot of simulated atoms (high K) to accurately preserve the tail and therefore the power-law somewhat limiting the utility of the result and disproving this would strengthen the paper in my view.    Overall I think this is a good paper that deserves to be on NIPS provided some editorial effort is given to the presentation.