Online Structure Learning for Sum-Product Networks with Gaussian Leaves
Sum-product networks have recently emerged as an attractive representation due to their dual view as a special type of deep neural network with clear semantics and a special type of probabilistic graphical model for which inference is always tractable. Those properties follow from some conditions (i.e., completeness and decomposability) that must be respected by the structure of the network. As a result, it is not easy to specify a valid sum-product network by hand and therefore structure learning techniques are typically used in practice. This paper describes the first {em online} structure learning technique for continuous SPNs with Gaussian leaves. We also introduce an accompanying new parameter learning technique.
The authors present an online learning method for learning the structure of sum-product networks. The algorithm assumes Gaussian coordinate-wise marginal distributions, and learns both parameters and structure online. The parameters are updated by a recursive procedure that reweights nodes in the network that most contribute to the likelihood of the current data point. The structure learning is done by either merging independent product Gaussian nodes into multivariate leaf nodes, or creating a mixture over the two nodes when the multivariate would be too large. The fact that the dataset is scaled to some larger datasets (in terms of the number of datapoints) is promising, although the number of variables is still quite small. Current benchmarks for tractable continuous density modeling with neural networks include the NICE and Real-NVP families of models, which can be scaled to both large numbers of datapoints and variables. Intractable methods like GAN, GenMMN, VAE have the same property. The main issue with this work for the ICLR audience is the use of mainly a set of SPN-specific datasets that are not used in the deep learning generative modeling literature. The use of GenMMN as a baseline is also not a good choice to bridge the gap to the neural community, as its Parzen-window based likelihood evaluation is not really meaningful. Better ways to evaluate the likelihood through annealed importance sampling are discussed in *On the Quantitative Analysis of Decoder-Based Generative Models* by Wu et al. I would recommend the use of a simple VAE type model to get a lower bound on the likelihood, or something like Real-NVP. Most neural network density models are scalable to large numbers of observations as well as instances, and it is not clear that this method scales well *horizontally* like this. Evaluating the feasibility of modeling something like MNIST would be interesting. SPNs have the strength that not only marginal but also various type of conditional queries are tractable, but performance on this is not evaluated or compared. One interesting application could be in imputation of unknown pixels or color channels in images, for which there is not currently a high-performing tractable model. Despite the disconnect from other ICLR generative modeling literature, the algorithm here seems simple and intuitive and convincingly works better than the previous state of the art for online SPN structure learning. I think VAE is a much better baseline for continuous data than GenMMN when attempting to compare to neural network approaches. Further, the sum-product network could actually be combined with such deep latent variable models as an observation model or posterior, which could be a very powerful combination. I would like it if these SPN models were better known by the ICLR probabilistic modeling community, but I do not know if this paper does enough to make them relevant. As with the other reviewers, I am not an expert on SPNs. However, this seems to be a simple and effective algorithm for online structure induction, and the scalability aspect is something that is important in much recent work in the learning of representations. I think it is good enough for publication, although I would prefer to see many of the above additions to more clearly bridge the gap with other literature in deep generative modeling.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Online Structure Learning for Sum-Product Networks with Gaussian Leaves
Sum-product networks have recently emerged as an attractive representation due to their dual view as a special type of deep neural network with clear semantics and a special type of probabilistic graphical model for which inference is always tractable. Those properties follow from some conditions (i.e., completeness and decomposability) that must be respected by the structure of the network. As a result, it is not easy to specify a valid sum-product network by hand and therefore structure learning techniques are typically used in practice. This paper describes the first {em online} structure learning technique for continuous SPNs with Gaussian leaves. We also introduce an accompanying new parameter learning technique.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors present an online learning method for learning the structure of sum-product networks. The algorithm assumes Gaussian coordinate-wise marginal distributions, and learns both parameters and structure online. The parameters are updated by a recursive procedure that reweights nodes in the network that most contribute to the likelihood of the current data point. The structure learning is done by either merging independent product Gaussian nodes into multivariate leaf nodes, or creating a mixture over the two nodes when the multivariate would be too large. The fact that the dataset is scaled to some larger datasets (in terms of the number of datapoints) is promising, although the number of variables is still quite small. Current benchmarks for tractable continuous density modeling with neural networks include the NICE and Real-NVP families of models, which can be scaled to both large numbers of datapoints and variables. Intractable methods like GAN, GenMMN, VAE have the same property. The main issue with this work for the ICLR audience is the use of mainly a set of SPN-specific datasets that are not used in the deep learning generative modeling literature. The use of GenMMN as a baseline is also not a good choice to bridge the gap to the neural community, as its Parzen-window based likelihood evaluation is not really meaningful. Better ways to evaluate the likelihood through annealed importance sampling are discussed in *On the Quantitative Analysis of Decoder-Based Generative Models* by Wu et al. I would recommend the use of a simple VAE type model to get a lower bound on the likelihood, or something like Real-NVP. Most neural network density models are scalable to large numbers of observations as well as instances, and it is not clear that this method scales well *horizontally* like this. Evaluating the feasibility of modeling something like MNIST would be interesting. SPNs have the strength that not only marginal but also various type of conditional queries are tractable, but performance on this is not evaluated or compared. One interesting application could be in imputation of unknown pixels or color channels in images, for which there is not currently a high-performing tractable model. Despite the disconnect from other ICLR generative modeling literature, the algorithm here seems simple and intuitive and convincingly works better than the previous state of the art for online SPN structure learning. I think VAE is a much better baseline for continuous data than GenMMN when attempting to compare to neural network approaches. Further, the sum-product network could actually be combined with such deep latent variable models as an observation model or posterior, which could be a very powerful combination. I would like it if these SPN models were better known by the ICLR probabilistic modeling community, but I do not know if this paper does enough to make them relevant. As with the other reviewers, I am not an expert on SPNs. However, this seems to be a simple and effective algorithm for online structure induction, and the scalability aspect is something that is important in much recent work in the learning of representations. I think it is good enough for publication, although I would prefer to see many of the above additions to more clearly bridge the gap with other literature in deep generative modeling.
assistant
The authors present an online learning method for learning the structure of sum-product networks. The algorithm assumes Gaussian coordinate-wise marginal distributions, and learns both parameters and structure online. The parameters are updated by a recursive procedure that reweights nodes in the network that most contribute to the likelihood of the current data point. The structure learning is done by either merging independent product Gaussian nodes into multivariate leaf nodes, or creating a mixture over the two nodes when the multivariate would be too large. The fact that the dataset is scaled to some larger datasets (in terms of the number of datapoints) is promising, although the number of variables is still quite small. Current benchmarks for tractable continuous density modeling with neural networks include the NICE and Real-NVP families of models, which can be scaled to both large numbers of datapoints and variables. Intractable methods like GAN, GenMMN, VAE have the same property. The main issue with this work for the ICLR audience is the use of mainly a set of SPN-specific datasets that are not used in the deep learning generative modeling literature. The use of GenMMN as a baseline is also not a good choice to bridge the gap to the neural community, as its Parzen-window based likelihood evaluation is not really meaningful. Better ways to evaluate the likelihood through annealed importance sampling are discussed in *On the Quantitative Analysis of Decoder-Based Generative Models* by Wu et al. I would recommend the use of a simple VAE type model to get a lower bound on the likelihood, or something like Real-NVP. Most neural network density models are scalable to large numbers of observations as well as instances, and it is not clear that this method scales well *horizontally* like this. Evaluating the feasibility of modeling something like MNIST would be interesting. SPNs have the strength that not only marginal but also various type of conditional queries are tractable, but performance on this is not evaluated or compared. One interesting application could be in imputation of unknown pixels or color channels in images, for which there is not currently a high-performing tractable model. Despite the disconnect from other ICLR generative modeling literature, the algorithm here seems simple and intuitive and convincingly works better than the previous state of the art for online SPN structure learning. I think VAE is a much better baseline for continuous data than GenMMN when attempting to compare to neural network approaches. Further, the sum-product network could actually be combined with such deep latent variable models as an observation model or posterior, which could be a very powerful combination. I would like it if these SPN models were better known by the ICLR probabilistic modeling community, but I do not know if this paper does enough to make them relevant. As with the other reviewers, I am not an expert on SPNs. However, this seems to be a simple and effective algorithm for online structure induction, and the scalability aspect is something that is important in much recent work in the learning of representations. I think it is good enough for publication, although I would prefer to see many of the above additions to more clearly bridge the gap with other literature in deep generative modeling.