Multiscale Dictionary Learning for Estimating Conditional Distributions
Nonparametric estimation of the conditional distribution of a response given highdimensional features is a challenging problem. It is important to allow not only the mean but also the variance and shape of the response density to change flexibly with features, which are massive-dimensional. We propose a multiscale dictionary learning model, which expresses the conditional response density as a convex combination of dictionary densities, with the densities used and their weights dependent on the path through a tree decomposition of the feature space. A fast graph partitioning algorithm is applied to obtain the tree decomposition, with Bayesian methods then used to adaptively prune and average over different sub-trees in a soft probabilistic manner. The algorithm scales efficiently to approximately one million features. State of the art predictive performance is demonstrated for toy examples and two neuroscience applications including up to a million features.
Summary: This paper addresses the problem of 
      conditional density estimation with a high dimensional input space (p 
      >> n), an important problems as most (if not all) current models for 
      nonparametric conditional density estimation do not scale to 
      high-dimensions. Moreover, datasets with high dimensional inputs but 
      relatively small sample sizes are becoming increasingly common. The model 
      for the conditional density f(y|x) is defined in three stages. First, a 
      tree structure is defined over the input space. Second, given the tree 
      structure, C_{j,k}, the k^th partition of the X space at scale j, is 
      mapped to a lower dimensional space. Third, a local model for the 
      conditional density at each node j,k is selected which only depends on x 
      through its lower dimensional representation. In practice the lower 
      dimension representation is the indicator function 1(x \in C_{j,k}) and 
      the local model is Gaussian, i.e. f_{j,k}(y|x)=N(m_{j,k}, s_{j,k}). 
      The inference algorithm is “partially Bayesian”, as an empirical Bayes 
      approach is used for the tree structure (which is estimated via the METIS 
      algorithm). Then, a Gibbs sampling algorithm is used for posterior 
      inference of the weights and parameters (m_{j,k}, s_{j,k}) of each node 
      indexed by j,k of the tree. Strengths: The authors develop a 
      novel method for high-dimensional conditional density estimation which 
      lacks any competitors. Performance is quite impressive when compared 
      with LASSO, CART, PC regression (even more so considering the added 
      flexibility of the model). Weaknesses: Poor notation and lack 
      of clear representation of the model for the conditional density. The 
      model is stated very vaguely, but then a more specific version is used. 
      The paper would be more readable if the focus was only on the specific 
      version of the model used in practiced and included a clear mathematical 
      description of the model. In equation (1), your model for the conditional 
      density given the tree structure tau, depends on some j,k. In particular, 
      f(y|x,tau) is a weighted average over the local conditional density of all 
      ancestors of j,k. How is this cut-off j,k selected? In equation (1), 
      your notation C_{j',k'} \in A_{j,k} does not match your definition of the 
      sets, as C_{j',k'} is a subset of the X space and A_{j,k} is a subset of 
      the integers. Throughout the text the conditional distribution 
      function is used when conditional density should be, leading to some 
      incorrect statements (examples are pg. line 86 and pg. 6 line 322). 
      The word approximately is used throughout the text, but it may be more 
      sensible to replace approximately with equals and state that these are 
      model assumptions. What does f_{y|x} represent? The entire collection 
      of conditional densities f_{y|x}=(f_{y|x}(\cdot|x), x\in X) or a specific 
      f_{y|x}(y|x)? Same comments for F_{Y|X} pg. 4 line 196, there is a 
      missing reference. pg. 4 equation (2), should proportional be replaced 
      with equals? And the last pi_{j,k}=1-\sum V_{j',k'} (where the sum is over 
      the ancestors of j,k). Otherwise it involves a normalization term which 
      may affect the full conditional of the V_{j,k}. pg. 5 line 239, the 
      notation k_j(x_i) is used but not introduced. pg. 5 line 242, Is the 
      full conditional of V_{j,k} correct? Because the likelihood seems to 
      involve a normalization term based on V_{j,k}. The paper by 
      Bhattacharya, Page, and Dunson (2013) Density Estimation and 
      Classification via Bayesian nonparametric learning of affine subspaces 
      could be an important reference/competitor to consider. Quality: 
      The work is supported by some impressive examples but the model lacks a 
      clear mathematical explanation. Clarity: I would suggest removing 
      the vague representation of the model and just focus on the specific model 
      used in practice. Notation needs some cleaning. Originality: 
      Similar to other regression techniques, the proposed model achieves 
      flexibility through a tree structure; however, its originality comes from 
      flexibility in modelling the entire conditional density not just the mean 
      function, which is achieved by allowing for local conditional densities at 
      each node of the tree. For inference in high-dimensions, they must take an 
      empirical Bayes approach with regards to the tree structure. 
      Significance: The results could potentially be very important as 
      the allow for conditional density estimation with high-dimensional inputs 
      and experimental results are impressive.