Flexible sampling of discrete data correlations without the marginal distributions
Learning the joint dependence of discrete variables is a fundamental problem in machine learning, with many applications including prediction, clustering and dimensionality reduction. More recently, the framework of copula modeling has gained popularity due to its modular parameterization of joint distributions. Among other properties, copulas provide a recipe for combining flexible models for univariate marginal distributions with parametric families suitable for potentially high dimensional dependence structures. More radically, the extended rank likelihood approach of Hoff (2007) bypasses learning marginal models completely when such information is ancillary to the learning task at hand as in, e.g., standard dimensionality reduction problems or copula parameter estimation. The main idea is to represent data by their observable rank statistics, ignoring any other information from the marginals. Inference is typically done in a Bayesian framework with Gaussian copulas, and it is complicated by the fact this implies sampling within a space where the number of constraints increases quadratically with the number of data points. The result is slow mixing when using off-the-shelf Gibbs sampling. We present an efficient algorithm based on recent advances on constrained Hamiltonian Markov chain Monte Carlo that is simple to implement and does not require paying for a quadratic cost in sample size.
This paper is concerned with learning Gaussian copulas 
      after the empirical cdf is used to transform the marginals of all data. 
      The challenge in this scenario seems to be linear constraints on the 
      sampling domain, the number of which increases quadratically with the 
      number of data points. The paper proposes a Hamiltonian Monte Carlo 
      sampler where the constraints are handled by the sample trajectory 
      "bouncing" off them. Empirical results are given, in particular showing 
      superior performance, for the same number of steps, against the Hoff 
      algorithm. The paper seems technically sound, although I have some 
      concerns regarding the empirical results. The presentation is fine, 
      although note minor points below. I have not seen an approach such as this 
      before and believe it to be original, although am not well-abreast of the 
      copula literature, and my confidence score reflects this. The approach 
      does seem quite computationally expensive, and the authors even admit that 
      their MATLAB implementation is not particularly fast, which leaves 
      question marks over potential impact. My concerns with the 
      empirical results are: * The HMC is only run for 1000 steps, while 
      the Hoff is run for 2000. The Hoff is clearly seen to make a few switches 
      after the 1000 step mark, and a reader might worry that the HMC method 
      could do the same. I presume this has been done because the HMC is much 
      more expensive to run (the authors say 50 times so in Footnote 7). I don't 
      doubt the final result, especially looking at the plots in the 
      supplementary material, but for thoroughness it would be good to see the 
      HMC run for 2000 steps, or a plot of posterior densities of each sample 
      that indicates that the jumps of the Hoff algorithm are not due to a 
      second mode, or a discussion point affirming that the posterior is 
      definitely unimodal (which I think is knowable in this case a priori). 
      * I'd like to see more discussion on the relative computational 
      expense of the HMC and Hoff methods. Footnote 7 makes an excuse, whereas 
      I'd rather see an explanation. Should I expect a good implementation of 
      HMC to be as fast as Hoff? At the moment, with HMC taking 50 times longer, 
      it may be fairer to compare HMC at iteration #50 with Hoff at iteration 
      #2500 in the bottom row of Figure 3. A few minor points on 
      presentation: * There's a citation, [12], given for the first time 
      in the conclusion. If this is important it should probably be introduced 
      as related work in the introduction. * The footnoting might be 
      considered excessive. I'd suggest that some of these could be moved into 
      the text without distracting the reader, especially 2, as it ends a 
      paragraph anyway, but 3, 5, and 7 as well. In fact 5 should be, as it 
      introduces notation, which is required knowledge rather than an aside, and 
      I think 7 also, as, in the context of comments on the empirical results 
      above, it's important to know that the HMC steps take a lot longer to run.