Conditional Random Fields via Univariate Exponential Families
Conditional random fields, which model the distribution of a multivariate response conditioned on a set of covariates using undirected graphs, are widely used in a variety of multivariate prediction applications. Popular instances of this class of models, such as categorical-discrete CRFs, Ising CRFs, and conditional Gaussian based CRFs, are not well suited to the varied types of response variables in many applications, including count-valued responses. We thus introduce a novel subclass of CRFs, derived by imposing node-wise conditional distributions of response variables conditioned on the rest of the responses and the covariates as arising from univariate exponential families. This allows us to derive novel multivariate CRFs given any univariate exponential distribution, including the Poisson, negative binomial, and exponential distributions. Also in particular, it addresses the common CRF problem of specifying "feature" functions determining the interactions between response variables and covariates. We develop a class of tractable penalized M -estimators to learn these CRF distributions from data, as well as a unified sparsistency analysis for this general class of CRFs showing exact structure recovery can be achieved with high probability.
This interesting paper contributes to the literature 
      on CRFs with covariate-dependent structure, making three separate and 
      interesting contributions: (a) (Theorem 1) generalizes the 
      classical and well-known correspondence between autoregressive Gaussian 
      models and multivariate Gaussians to arbitrary. The interesting and 
      subtle bit here is the dependence of equations (2) and (3) on X. We 
      started in (1) by allowing the autoregressive weights to depend 
      arbitrarily on X and Y\s, but by making an additional CRF assumption on 
      P(Y|X) we find that these weights factorize into a terms that depends 
      on X and the sufficient statistics of Y_s's neighbours. (b) 
      Section 2.1: If we assume an exponential family and graphical model over X 
      in addition to our CRF P(Y|X), this implies additional structure in the 
      model parameters of the CRF. This is a neat idea. (c) The 
      structure from (b) allows one to use a particular l1 regularizer for 
      structure discovery, for which we can provide guarantees about learning 
      error and structure recovery. Although (a) and (c) do appear to 
      follow relatively closely the previous results from Yang et al about MRFs, 
      (b) certainly seems more new, as it is specific to conditional models. 
      One (perhaps too ambitious) question that I had was regarding the 
      marginal distribution P(Y_s | X). If the node conditionals are Gaussian, 
      then of course these marginals are Gaussian; similarly if the node 
      conditions are fully-parameterized discrete distributions, then the node 
      marginals are also in the same exponential family. I don't suppose that 
      this property would hold for other exponential families? I have 
      only minor suggestions for improvement: * I'm not sure that the 
      Introduction sells the contribution of the paper as clearly as it could, 
      as it seems to focus on having different exponential families in the 
      autoregressive structure, where it seems to me the real modelling 
      contribution is the interplay among the autoregressive weights and the 
      sufficient statistics of X. * The real data example in Section 4 
      seems like it is probably a nice example of the modelling benefits of 
      having covariate-dependent structure, although this is almost impossible 
      to understand for readers who are not already familiar with the biology in 
      this example. It would be nice to expand the discussion of these results 
      for readers coming from a more general ML background, althought I 
      understand that this could be difficult given space constraints. * 
      Theorem (1): The statement of the theorem is fairly long, and the phrase 
      "These assertions are consistent" is a bit less precise than I'd perhaps 
      like. I wonder if it's possible to introduce some terminology that 
      makes clauses (a) and (b) shorter and therefore the theorem easier to 
      understand. * Equation (2): I am assuming that if the functions 
      B_s return a vector, then the product should be interpreted elementwise? 
      * line 157: B_s(Y_s) = Y_s. Surely Y_s^2 needs to be a sufficient 
      statistic somehow? * line 298: This is a *pseudo*-likelihood, yes? 
      * line 316: Are the constants M_1 and M_2 computable? Is there a 
      simple expression for them? Are these constraints always satisfiable?