Parallel Sampling of HDPs using Sub-Cluster Splits
We develop a sampling technique for Hierarchical Dirichlet process models. The parallel algorithm builds upon [1] by proposing large split and merge moves based on learned sub-clusters. The additional global split and merge moves drastically improve convergence in the experimental results. Furthermore, we discover that cross-validation techniques do not adequately determine convergence, and that previous sampling methods converge slower than were previously expected.
Summary: The paper develops a MCMC algorithm for the Hierarchical Dirichlet process (HDP) that accommodates splits and merges and can be parallelized. Issues with respect to developing effective splits are addressed and controlled experiments demonstrating the algorithm's effectiveness are presented.This is a well written, technically sound paper. The primary contribution lies in extending the DP sub cluster framework of [1] to HDPs. Although, this is somewhat incremental, the innovations required to transition the ideas in [1] to HDP inference are significant enough to warrant publication. Detailed Comments:1) Global split/merge moves: I like these moves and it is certainly believable that to escape certain "local optima" moves that distribute a topic's mass globally are required. However, given the significant extra cost of having to loop through all data instances it is important to clearly demonstrate the benefit of incorporating such moves. To this end, the experiments with and without the global moves on the toy bars dataset are not convincing. The authors do mention that they notice significant improvements on the larger real world datasets. I would like to see an additional plot added to Figures 4 and 5 from a handicapped version of the algorithm that uses only local moves.2) It appears that at each iteration, the topic to be globally split is being chosen uniformly at random. Perhaps a more data driven proposal could be utilized here. For example, the negative cross entropy measure (C) defined in lines 375-376 provides a nice heuristic for determining "confused" topics. A topic, many of whose words have high probability under other topics, could be  selected with higher probability for a split. 3) The issue of approximating the Hastings ratio for the merge moves is somewhat confusing. Are the MCMC guarantees maintained in spite of the approximation? 4) Is there a reason no experimental comparison against Wang and Blei's SAMS sampler is presented? The code is publicly available and should really be compared against.