Inference and Introspection in Deep Generative Models of Sparse Data
Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model.
This paper presents a small trick to improve the model quality of variational autoencoders (further optimizing the ELBO while initializing it from the predictions of the q network, instead of just using those directly) and the idea of using Jacobian vectors to replace simple embeddings when interpreting variational autoencoders. The idea of the Jacobian as a natural replacement for embeddings is interesting, as it does seem to cleanly generalize the notion of embeddings from linear models. It*d be interesting to see comparisons with other work seeking to provide context-specific embeddings, either by clustering or by smarter techniques (like Neelakantan et al, Efficient non-parametric estimation of multiple embeddings per word in vector space, or Chen et al A Unified Model for Word Sense Representation and Disambiguation). With the evidence provided in the experimental section of the paper it*s hard to be convinced that the Jacobian of VAE-generated embeddings is substantially better at being context-sensitive than prior work. Similarly, the idea of further optimizing the ELBO is interesting but not fully explored. It*s unclear, for example, what is the tradeoff between the complexity of the q network and steps further optimizing the ELBO, in terms of compute versus accuracy. Overall the ideas in this paper are good but I*d like to see them a little more fleshed out.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Inference and Introspection in Deep Generative Models of Sparse Data
Deep generative models such as deep latent Gaussian models (DLGMs) are powerful and popular density estimators. However, they have been applied almost exclusively to dense data such as images; DLGMs are rarely applied to sparse, high-dimensional integer data such as word counts or product ratings. One reason is that the standard training procedures find poor local optima when applied to such data. We propose two techniques that alleviate this problem, significantly improving our ability to fit DLGMs to sparse, high-dimensional data. Having fit these models, we are faced with another challenge: how to use and interpret the representation that we have learned? To that end, we propose a method that extracts distributed representations of features via a simple linearization of the model.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper presents a small trick to improve the model quality of variational autoencoders (further optimizing the ELBO while initializing it from the predictions of the q network, instead of just using those directly) and the idea of using Jacobian vectors to replace simple embeddings when interpreting variational autoencoders. The idea of the Jacobian as a natural replacement for embeddings is interesting, as it does seem to cleanly generalize the notion of embeddings from linear models. It*d be interesting to see comparisons with other work seeking to provide context-specific embeddings, either by clustering or by smarter techniques (like Neelakantan et al, Efficient non-parametric estimation of multiple embeddings per word in vector space, or Chen et al A Unified Model for Word Sense Representation and Disambiguation). With the evidence provided in the experimental section of the paper it*s hard to be convinced that the Jacobian of VAE-generated embeddings is substantially better at being context-sensitive than prior work. Similarly, the idea of further optimizing the ELBO is interesting but not fully explored. It*s unclear, for example, what is the tradeoff between the complexity of the q network and steps further optimizing the ELBO, in terms of compute versus accuracy. Overall the ideas in this paper are good but I*d like to see them a little more fleshed out.
assistant
This paper presents a small trick to improve the model quality of variational autoencoders (further optimizing the ELBO while initializing it from the predictions of the q network, instead of just using those directly) and the idea of using Jacobian vectors to replace simple embeddings when interpreting variational autoencoders. The idea of the Jacobian as a natural replacement for embeddings is interesting, as it does seem to cleanly generalize the notion of embeddings from linear models. It*d be interesting to see comparisons with other work seeking to provide context-specific embeddings, either by clustering or by smarter techniques (like Neelakantan et al, Efficient non-parametric estimation of multiple embeddings per word in vector space, or Chen et al A Unified Model for Word Sense Representation and Disambiguation). With the evidence provided in the experimental section of the paper it*s hard to be convinced that the Jacobian of VAE-generated embeddings is substantially better at being context-sensitive than prior work. Similarly, the idea of further optimizing the ELBO is interesting but not fully explored. It*s unclear, for example, what is the tradeoff between the complexity of the q network and steps further optimizing the ELBO, in terms of compute versus accuracy. Overall the ideas in this paper are good but I*d like to see them a little more fleshed out.