Sparse Bayesian structure learning with dependent relevance determination prior
In many problem settings, parameter vectors are not merely sparse, but dependent in such a way that non-zero coefficients tend to cluster together. We refer to this form of dependency as "region sparsity". Classical sparse regression methods, such as the lasso and automatic relevance determination (ARD), model parameters as independent a priori, and therefore do not exploit such dependencies. Here we introduce a hierarchical model for smooth, region-sparse weight vectors and tensors in a linear regression setting. Our approach represents a hierarchical extension of the relevance determination framework, where we add a transformed Gaussian process to model the dependencies between the prior variances of regression weights. We combine this with a structured model of the prior variances of Fourier coefficients, which eliminates unnecessary high frequencies. The resulting prior encourages weights to be region-sparse in two different bases simultaneously. We develop efficient approximate inference methods and show substantial improvements over comparable methods (e.g., group lasso and smooth RVM) for both simulated and real datasets from brain imaging.
I appreciate the thorough author feedback and find the authors' arguments mostly compelling.  I have decided to increase my review score to be above the acceptance threshold under the understanding that the authors will clarify the technical concerns in a final paper and expand on the bibliography.  I am still not convinced that the covariance function is positive semi-definite for inputs with dimension greater than one.  I understand that as the inputs correspond to the position of the weights they will generally will be of low dimension.  However, it is much more interesting to allow the weights to be located in 2D (images), 3D (videos) or even a higher dimensional latent space.  As a simple remedy, the authors could make the covariance be the product over dimensions as in [11] or they could restrict the dimensionality of the inputs to be one.  Otherwise, I would like more justification in the manuscript or in an appendix that the proposed covariance is PSD.------------------This paper explores the problem of linear regression in the scenario where many of the weights are expected to be zero.  The innovation of the paper is the development of a hierarchical Bayesian model for which this sparsity follows a spatial pattern, the sparsity of the weights is dependent on the relative spatial location of each weight.  The sparsity pattern is achieved through placing a Gaussian process prior on the regression weights.  Specifically, the authors place a GP prior, conditioned on the spatial location of the weights, over a latent variable governing the diagonal of a covariance matrix.  This is composed with a smoothness covariance in such a way that the latent variables govern the magnitude of the eigenvalues.  After fitting the hyperparameters of the model, the authors prune the weights corresponding to small eigenvalues of the covariance, achieving the desired sparsity.The paper is well written, structured and easy to understand.  The authors do a good job of motivating the problem and sparse regression is certainly a topic of interest to the NIPS community.  The model seems novel and interesting and the experiments, although very limited, do demonstrate the quality of the approach.  There are, however, some concerns with the methodology and the correctness of the model and the empirical evaluation is not exhaustive enough to be compelling.Although the authors start with a neat model, the approach seems to turn into a rather heuristic procedure due to scalability, tractability and numerical issues.  Although I empathize with the authors, the Bayesian model becomes an approximation to an approximation of the original model.  The paper seems like a very good start to an interesting model, but the approach seems a little suspect (i.e. PSD-ness, pruning weights corresponding to small eigenvalues, inverting with the SVD due to a singular covariance, various approximations) and thus I would like significantly more justification of this method both in the text and empirically.A major concern is the chosen covariance function (equation 5).  Specifically, I doubt that it is positive definite and I find it odd in general.  Why would one have a periodic covariance over the location of the weights?  This seems strange as some closer weights will be anticorrelated while farther weights will be correlated.  I think this is only positive definite if the distance between the location of the weights i and j is bounded between [pi/2, -pi/2].  Perhaps this is a reason for the numerical instability that is referred to on line 208?  The authors must justify that this covariance function is a reasonable choice (i.e. show that it is PSD) and provide some reasoning.The authors should expand significantly on the related work and references.  There is a wealth of literature about structured sparse regression from recent work in submodularity and convex optimization (see e.g. Bach, http://arxiv.org/abs/1111.6453), and in the machine learning literature in general (e.g. Huang et. al http://jmlr.org/papers/volume12/huang11b/huang11b.pdf).  There is plenty of space on the references page to add these.Radford Neal and David Mackay should really be referenced for automatic relevance determination (ARD).In the abstract, the authors suggest that the Gaussian process prior is conditioned on the Fourier coefficients of the weights.  I'm not sure I understand how the Fourier coefficients come into play here and would appreciate some clarification.There is no analysis done on the impact of the various approximations on the posterior of the model.  Specifically, how is the predictive variance of the model affected with respect to the approximations?  Do they negatively affect the uncertainty of the model (I think they probably do significantly).  I find it strange that the smooth RVM is left out of the final experiment.  This should be included to make the empirical analysis compelling.  However, at least some of the wealth of existing structured sparsity work should also be compared to.  Even if that work is not Bayesian, it give a better perspective on how difficult the problem is.  Finally, it seems to me that Gaussian processes should really be included in the comparison.  Why are these left out?  It doesn't seem that the proposed model scales beyond what a GP can model.