Variational Message Passing with Structured Inference Networks
Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods.
 The paper seems to be significant since it integrates PGM inference with deep models. Specifically, the idea is to use the structure of the PGM to perform efficient inference. A variational message passing approach is developed which performs natural-gradient updates for the PGM part and stochastic gradient updates for the deep model part. Performance comparison is performed with an existing approach that does not utilize the PGM structure for inference. The paper does a good job of explaining the challenges of inference, and provides a systematic approach to integrating PGMs with deep model updates. As compared to the existing approach where the PGM parameters must converge before updating the DNN parameters, the proposed architecture does not require this, due to the re-parameterization which is an important contribution. The motivation of the paper, and the description of its contribution as compared to existing methods can be improved. One of the main aspects it seems is generality, but the encodings are specific to 2 types PGMs. Can this be generalized to arbitrary PGM structures? How about cases when computing Z is intractable? Could the proposed approach be adapted to such cases. I was not very sure as to why the proposed method is more general than existing approaches. Regarding the experiments, as mentioned in the paper the evaluation is performed on two fairly small scale datasets. the approach shows that the proposed methods converge faster than existing methods. However, I think there is value in the approach, and the connection between variational methods with DNNs is interesting.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Variational Message Passing with Structured Inference Networks
Recent efforts on combining deep models with probabilistic graphical models are promising in providing flexible models that are also easy to interpret. We propose a variational message-passing algorithm for variational inference in such models. We make three contributions. First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE). Second, we establish conditions under which such inference networks enable fast amortized inference similar to VAE. Finally, we derive a variational message passing algorithm to perform efficient natural-gradient inference while retaining the efficiency of the amortized inference. By simultaneously enabling structured, amortized, and natural-gradient inference for deep structured models, our method simplifies and generalizes existing methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper seems to be significant since it integrates PGM inference with deep models. Specifically, the idea is to use the structure of the PGM to perform efficient inference. A variational message passing approach is developed which performs natural-gradient updates for the PGM part and stochastic gradient updates for the deep model part. Performance comparison is performed with an existing approach that does not utilize the PGM structure for inference. The paper does a good job of explaining the challenges of inference, and provides a systematic approach to integrating PGMs with deep model updates. As compared to the existing approach where the PGM parameters must converge before updating the DNN parameters, the proposed architecture does not require this, due to the re-parameterization which is an important contribution. The motivation of the paper, and the description of its contribution as compared to existing methods can be improved. One of the main aspects it seems is generality, but the encodings are specific to 2 types PGMs. Can this be generalized to arbitrary PGM structures? How about cases when computing Z is intractable? Could the proposed approach be adapted to such cases. I was not very sure as to why the proposed method is more general than existing approaches. Regarding the experiments, as mentioned in the paper the evaluation is performed on two fairly small scale datasets. the approach shows that the proposed methods converge faster than existing methods. However, I think there is value in the approach, and the connection between variational methods with DNNs is interesting.
assistant
 The paper seems to be significant since it integrates PGM inference with deep models. Specifically, the idea is to use the structure of the PGM to perform efficient inference. A variational message passing approach is developed which performs natural-gradient updates for the PGM part and stochastic gradient updates for the deep model part. Performance comparison is performed with an existing approach that does not utilize the PGM structure for inference. The paper does a good job of explaining the challenges of inference, and provides a systematic approach to integrating PGMs with deep model updates. As compared to the existing approach where the PGM parameters must converge before updating the DNN parameters, the proposed architecture does not require this, due to the re-parameterization which is an important contribution. The motivation of the paper, and the description of its contribution as compared to existing methods can be improved. One of the main aspects it seems is generality, but the encodings are specific to 2 types PGMs. Can this be generalized to arbitrary PGM structures? How about cases when computing Z is intractable? Could the proposed approach be adapted to such cases. I was not very sure as to why the proposed method is more general than existing approaches. Regarding the experiments, as mentioned in the paper the evaluation is performed on two fairly small scale datasets. the approach shows that the proposed methods converge faster than existing methods. However, I think there is value in the approach, and the connection between variational methods with DNNs is interesting.