Tree-Structured Variational Autoencoder
Many kinds of variable-sized data we would like to model contain an internal hierarchical structure in the form of a tree, including source code, formal logical statements, and natural language sentences with parse trees. For such data it is natural to consider a model with matching computational structure. In this work, we introduce a variational autoencoder-based generative model for tree-structured data. We evaluate our model on a synthetic dataset, and a dataset with applications to automated theorem proving. By learning a latent representation over trees, our model can achieve similar test log likelihood to a standard autoregressive decoder, but with the number of sequentially dependent computations proportional to the depth of the tree instead of the number of nodes in the tree.
This paper introduces a novel extension of the variational autoencoder to arbitrary tree-structured outputs. Experiments are conducted on a synthetic arithmetic expression dataset and a first-order logic proof clause dataset in order to evaluate its density modeling performance. Pros: + The paper is clear and well-written. + The tree-structure definition is sufficiently complete to capture a wide variety of tree types found in real-world situations. + The tree generation and encoding procedure is elegant and well-articulated. + The experiments, though limited in scope, are relatively thorough. The use of IWAE to obtain a better estimate of log likelihoods is a particularly nice touch. Cons: - The performance gain over a baseline sequential model is marginal. - The experiments are limited in scope, both in the datasets considered and in the evaluation metrics used to compare the model with other approaches. Specifically: (a) there is only one set of results on a real-world dataset and in that case the proposed model performs worse than the baseline, and (b) there is no evaluation of the learned latent representation with respect to other tasks such as classification. - The ability of the model to generate trees in time proportional to the depth of the tree is proposed as a benefit of the approach, though this is not empirically validated in the experiments. The procedures to generate and encode trees are clever in their repeated use of common operations. The weight sharing and gating operations seem important for this model to perform well but it is difficult to assess their utility without an ablation (in Table 1 and 2 these modifications are not evaluated side-by-side). Experiments in another domain (such as modeling source code, or parse trees conditioned on a sentence) would help in demonstrating the utility of this model. Overall the model seems promising and applicable to a variety of data but the lack of breadth in the experiments is a concern. * Section 3.1: *We distinguish three types* => two * Section 3.6: The exposition of the variable-sized latent state is slightly confusing because the issue of how many z*s to generate is not discussed. * Section 4.2-4.3: When generating the datasets, did you verify that the test set is disjoint from the training set? * Table 1: Is there a particular reason why the variable latent results are missing for the depth 11 trees?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Tree-Structured Variational Autoencoder
Many kinds of variable-sized data we would like to model contain an internal hierarchical structure in the form of a tree, including source code, formal logical statements, and natural language sentences with parse trees. For such data it is natural to consider a model with matching computational structure. In this work, we introduce a variational autoencoder-based generative model for tree-structured data. We evaluate our model on a synthetic dataset, and a dataset with applications to automated theorem proving. By learning a latent representation over trees, our model can achieve similar test log likelihood to a standard autoregressive decoder, but with the number of sequentially dependent computations proportional to the depth of the tree instead of the number of nodes in the tree.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper introduces a novel extension of the variational autoencoder to arbitrary tree-structured outputs. Experiments are conducted on a synthetic arithmetic expression dataset and a first-order logic proof clause dataset in order to evaluate its density modeling performance. Pros: + The paper is clear and well-written. + The tree-structure definition is sufficiently complete to capture a wide variety of tree types found in real-world situations. + The tree generation and encoding procedure is elegant and well-articulated. + The experiments, though limited in scope, are relatively thorough. The use of IWAE to obtain a better estimate of log likelihoods is a particularly nice touch. Cons: - The performance gain over a baseline sequential model is marginal. - The experiments are limited in scope, both in the datasets considered and in the evaluation metrics used to compare the model with other approaches. Specifically: (a) there is only one set of results on a real-world dataset and in that case the proposed model performs worse than the baseline, and (b) there is no evaluation of the learned latent representation with respect to other tasks such as classification. - The ability of the model to generate trees in time proportional to the depth of the tree is proposed as a benefit of the approach, though this is not empirically validated in the experiments. The procedures to generate and encode trees are clever in their repeated use of common operations. The weight sharing and gating operations seem important for this model to perform well but it is difficult to assess their utility without an ablation (in Table 1 and 2 these modifications are not evaluated side-by-side). Experiments in another domain (such as modeling source code, or parse trees conditioned on a sentence) would help in demonstrating the utility of this model. Overall the model seems promising and applicable to a variety of data but the lack of breadth in the experiments is a concern. * Section 3.1: *We distinguish three types* => two * Section 3.6: The exposition of the variable-sized latent state is slightly confusing because the issue of how many z*s to generate is not discussed. * Section 4.2-4.3: When generating the datasets, did you verify that the test set is disjoint from the training set? * Table 1: Is there a particular reason why the variable latent results are missing for the depth 11 trees?
assistant
This paper introduces a novel extension of the variational autoencoder to arbitrary tree-structured outputs. Experiments are conducted on a synthetic arithmetic expression dataset and a first-order logic proof clause dataset in order to evaluate its density modeling performance. Pros: + The paper is clear and well-written. + The tree-structure definition is sufficiently complete to capture a wide variety of tree types found in real-world situations. + The tree generation and encoding procedure is elegant and well-articulated. + The experiments, though limited in scope, are relatively thorough. The use of IWAE to obtain a better estimate of log likelihoods is a particularly nice touch. Cons: - The performance gain over a baseline sequential model is marginal. - The experiments are limited in scope, both in the datasets considered and in the evaluation metrics used to compare the model with other approaches. Specifically: (a) there is only one set of results on a real-world dataset and in that case the proposed model performs worse than the baseline, and (b) there is no evaluation of the learned latent representation with respect to other tasks such as classification. - The ability of the model to generate trees in time proportional to the depth of the tree is proposed as a benefit of the approach, though this is not empirically validated in the experiments. The procedures to generate and encode trees are clever in their repeated use of common operations. The weight sharing and gating operations seem important for this model to perform well but it is difficult to assess their utility without an ablation (in Table 1 and 2 these modifications are not evaluated side-by-side). Experiments in another domain (such as modeling source code, or parse trees conditioned on a sentence) would help in demonstrating the utility of this model. Overall the model seems promising and applicable to a variety of data but the lack of breadth in the experiments is a concern. * Section 3.1: *We distinguish three types* => two * Section 3.6: The exposition of the variable-sized latent state is slightly confusing because the issue of how many z*s to generate is not discussed. * Section 4.2-4.3: When generating the datasets, did you verify that the test set is disjoint from the training set? * Table 1: Is there a particular reason why the variable latent results are missing for the depth 11 trees?