Learning Structured Densities via Infinite Dimensional Exponential Families
Learning the structure of a probabilistic graphical models is a well studied problem in the machine learning community due to its importance in many applications. Current approaches are mainly focused on learning the structure under restrictive parametric assumptions, which limits the applicability of these methods. In this paper, we study the problem of estimating the structure of a probabilistic graphical model without assuming a particular parametric model. We consider probabilities that are members of an infinite dimensional exponential family [4], which is parametrized by a reproducing kernel Hilbert space (RKHS) H and its kernel k. One difficulty in learning nonparametric densities is the evaluation of the normalizing constant. In order to avoid this issue, our procedure minimizes the penalized score matching objective [10,11]. We show how to efficiently minimize the proposed objective using existing group lasso solvers. Furthermore, we prove that our procedure recovers the graph structure with high-probability under mild conditions. Simulation studies illustrate ability of our procedure to recover the true graph structure without the knowledge of the data generating process.
I have read the authors' rebuttal and it did not change my opinion of the paper. -------- 1 Summary of the paper:This paper proposes to learn the structure of a kernel-based graphical model encoding a joint probability distribution (probabilities belonging to an infinite dimensional exponential family[1]). By using score-matching as the objective of the learning procedure, the normalisation constant does not need to be estimated, which is scalable and sufficient to learn the structure. Moreover, because of the kernel-based parameterisation, the optimisation problem reduces to an instance of the group lasso. The structure recovered by the algorithm is asymptotically consistent. The method proposed is compared to the graphical lasso to estimate a gaussian distribution and to both glasso and an algorithm knowing the model family on a paranormal distribution[1] S. Canu and A. Smola. Kernel methods and the exponential family. Neurocomputing, 69(7):714-720, 2006.2 Summary of the review: While I did not verify the theoretical analysis provided in the paper and the experiments could be explained better, I find the paper well organized, the contributions reasonably clear, the idea very interesting and potentially impactful. 3 Qualitative evaluation Quality: medium + extensive theoretical analysis (not checked fully, seems sound) +- small scale empirical evaluation - no discussion or empirical evaluation of how to choose the kernel parameters. Is a grid-search + cross-validation feasible and sufficient? How sensible is the result to the parameters chosen? - no code providedClarity: low + well organised paper, clear language +- lots of mathematical developments that are not easy to follow, especially with notations not necessarily introduced in the main paper.- the parameters of the second model learned in the experiments are not provided.Originality: good + I am not familiar with kernel-based methods or kernel-based graphical models, but I could not find really similar works beyond the ones cited.+ significant development with respect to previously published cited work - still, other works using kernel with PGMs exists. It could be interesting to relate the paper under review to them, for the informed reader like myself. For example, a quick search on google yielded the following papers: "Nonparametric Tree Graphical Models via Kernel Embeddings", "Kernel Conditional Random Fields: Representation and Clique Selection", "Learning Graphical Models with Mercer Kernels". Significance: good + structure learning for non-gaussian variable is an important open problem + I find the proposed idea very interestingImpact: 24 Other Comments:Details: . I understand you focus on the structure estimation problem, but can the normalisation constant be estimated easily enough when the model has been learned if one also wants the density? Are there some restricted families of kernel that make it possible? . I think section headers should not be all capitalized. . I am not familiar with RKHS. So I could not easily go from l204 to l207. I would suggest deriving it in the supplementary material. . a table of all notations in the supplementary material would make the paper a bit easier to follow. . l 319: why is it Omega(f)^2? (was Omega(f) in equation 8) . experiment 2: what is the dimensionality of the problem? What are the parameter values? . it would be nice to have the expression for D, E and F in theorem 3.2, for example in the appendix.typos: . l25: evaluation of --> the evaluation of . l53: "the" structure (lots of similar missing "the" in the paper - though I'm not a native speaker.) . l 76: integrable . l86: H_2?. l233: S^c not defined . l323: Omega*_{S^c} not defined