A simple example of Dirichlet process mixture inconsistency for the number of components
For data assumed to come from a finite mixture with an unknown number of components, it has become common to use Dirichlet process mixtures (DPMs) not only for density estimation, but also for inferences about the number of components. The typical approach is to use the posterior distribution on the number of clusters -that is, the posterior on the number of components represented in the observed data. However, it turns out that this posterior is not consistent -it does not concentrate at the true number of components. In this note, we give an elementary proof of this inconsistency in what is perhaps the simplest possible setting: a DPM with normal components of unit variance, applied to data from a "mixture" with one standard normal component. Further, we show that this example exhibits severe inconsistency: instead of going to 1, the posterior probability that there is one cluster converges (in probability) to 0.
The title of this paper is much like the paper itself: 
      to-the-point, descriptive, and readable. "A simple example of Dirichlet 
      process mixture inconsistency for the number of components" delivers on 
      its promise by providing two easy-to-understand demonstrations of the 
      severity of the problem of using Dirichlet process mixtures to estimate 
      the number of components in a mixture model. The authors start by 
      demonstrating that making such a component-cardinality estimate is 
      widespread in the literature (and therefore a problem deserving of 
      interest), briefly describe the Dirichlet process mixture (DPM) model 
      (with particular emphasis on the popular normal likelihood case), and then 
      demonstrate with a simple single-component mixture example how poorly 
      estimation of component cardinality can go (their convincing answer: very 
      poorly). Not only was the paper enjoyable to read but, 
      refreshingly, didn't try to fit 20 pages of material into an 8 page limit. 
      One potential criticism of this paper is that this result should 
      be well-known in some sense in the community. It is a widely cited result 
      that the number of components of a DPM grows logarithmically with the size 
      of the data (and indeed the popularity of the Pitman-Yor process extension 
      of the DPM is due to the different way in which the number of components 
      grows: as an almost sure power of the size of the data rather than merely 
      logarithmically). However, this paper is still necessary for two key 
      reasons. (1) Existing results deal only obliquely with consistency in 
      component cardinality, and this paper seems to be the first to prove 
      component-cardinality inconsistency directly. (2) The authors make a case 
      that the community, in spite of existing results, uses the DPM to estimate 
      the ``true'' number of components in a data set. Since there are cases 
      where the number of components in a mixture can indeed be expected to grow 
      with the size of the data (cf. the HDP extension to LDA for topic 
      modeling), the authors could make an even more compelling argument by 
      highlighting in text and via their citations those cases in the literature 
      where other authors use the DPM while assuming a fixed (i.e., not varying 
      with the number of data points) number of clusters in a data set of any 
      size. The authors might, in particular, make the distinction between the 
      number of components in the DPM generative model (infinite, as on line 38, 
      page 1) and the number of components "active" in a given data set. It is 
      not clear to me that using the DPM to estimate the active number of 
      components for a data set of size N, while assuming that a data set of 
      greater size would exhibit more components, is immediately problematic; 
      the real problem seems to be estimating a fixed, underlying number of 
      components. Some care in describing this subtlety might be useful. 
      Perhaps my one major concern is that at a number of times in the 
      paper the authors make what appear to be unsupported claims by referring 
      to seemingly already-completed future work---that is, work that is not in 
      the current paper or cited. E.g., on line 46, page 1, "as we will show 
      elsewhere" and line 66, page 2 "Further details will be provided 
      elsewhere", and even the extensions to general alpha in Proposition 3.1 
      and Theorem 4.1. It seems inappropriate to state results that are not 
      proved in the current paper or elsewhere in existing published 
      literature---except perhaps as proposed "future work" in the conclusion. 
      There is no way to judge the merits of these results until they are given 
      in full. Moreover, I think the paper stands on its own merits, and these 
      further results are not strictly necessary (though they might be nice 
      bonuses). Here are some more details: * For page 1, line 46, I suggest 
      removing the sentence or giving the full result; the counterexamples 
      provided either stand on their own or they don't. * Page 2, lines 
      64-66 make some interesting claims but again do not seem to be supported 
      by any citations or work in the current paper. One option is to simply add 
      the sampler (and some timing and experimental results) to this paper. 
      Another is to delete the sentences or move them to a "future work" 
      discussion. I agree that it's useful to point out that there are practical 
      alternatives to the DPM. But part of what's problematic about these 
      unsupported claims is that the reader can't tell whether many types of 
      inference (including approximations that are not MCMC) are available or 
      evaluate how the two MCMC samplers compare in speed, ease of use, etc. 
      * In Proposition 3.1 and Theorem 4.1, I suggest providing a full proof 
      of the general alpha case in the supplementary material (or in the main 
      text if it is indeed "trivial" as stated on page 4, line 167). I 
      also have a number of minor comments: * Page 2, line 72: Perhaps say a 
      bit more explicitly about why the DPM cannot be used as a heuristic. I 
      assume the idea here is that, given a fixed number of underlying 
      components, we expect the same amount of heterogeneity across a large 
      range of data cardinalities but can also expect the DPM to return very 
      different estimates of heterogeneity as data-size changes in these cases. 
      * Page 2, line 101: Introduce alpha in the text as a hyperparameter of 
      the model before using it. Notation for the CRP is inconsistent across 
      different disciplines, so it would help to name alpha as the 
      "concentration parameter" and give its allowable range of values. * 
      Page 2: This is a nice, careful treatment of ordered vs. unordered 
      partitions. * Page 3, line 126: The "component" vs "cluster" 
      distinction is somewhat unclear. Consider drawing upon notation to 
      clarify. * Page 4, line 107: Note that X_1, X_2, ... ~ N(0,1) is a 
      different assumption than in the statement of Proposition 3.1. I believe 
      the proof given here holds for the more general case of the Proposition 
      3.1 statement though. Interestingly, it seems to hold when X_1, X_2, etc 
      are iid from a single point mass. If indeed true, that seems like a 
      compelling special case that may be worth mentioning. * Page 6, line 
      304: Consider making this remark about p(A) = 1/n in this special case the 
      first time it is used instead (on page 4, line 192). ======== 
      Update after rebuttal: I have read the author response, which 
      addressed my few and minor concerns. I look forward to reading the final 
      product.