An information-theoretic analysis of deep latent-variable models
We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.
 - I think that VAEs are rather forced to be interpreted from an information theoretic point of view for the sake of it, rather than for the sake of a clear and unequivocal contribution from the perspective of VAEs and latent-variable models themselves. How is that useful for a VAE? - "The left vertical line corresponds to the zero rate setting. ...": All these limits are again from an information theoretic point of view and no formulation nor demonstration is provided on how this can actually be as useful. As mentioned earlier in the paper, there are well-known problems with taking this information theory perspective, e.g. difficulties in estimating MI values, etc. - Breaking (some of) the long sentences and paragraphs in page 3 with an unequivocal mathematical formulation would smooth the flow a bit. - "(2) an upper bound that measures the rate, or how costly it is to transmit information about the latent variable.": I am not entirely sure about this one and why it is massively important to be compromised against the obviously big first term. - Toy Model experiment: I do not see any indication of how this is not just a lucky catch and that VAEs consistently suffer from a problem leading to such effect. - Section 5: "can shed light on many different models and objectives that have been proposed in the literature... ": Again the contribution aspect is not so clear through the word "shed light". Minor - Although apparently VAEs represent the main and most influential latent-variable model example, I think switching too much between citing them as VAEs and then as latent-variable models in general was a bit confusing. I propose mentioning in the beginning (as happened) that VAEs are the seminal example of latent-variable models and then going on from this point onwards with VAEs without too much alternation between latent-variable models and VAEs. - page 8: "as show"
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
An information-theoretic analysis of deep latent-variable models
We present an information-theoretic framework for understanding trade-offs in unsupervised learning of deep latent-variables models using variational inference. This framework emphasizes the need to consider latent-variable models along two dimensions: the ability to reconstruct inputs (distortion) and the communication cost (rate). We derive the optimal frontier of generative models in the two-dimensional rate-distortion plane, and show how the standard evidence lower bound objective is insufficient to select between points along this frontier. However, by performing targeted optimization to learn generative models with different rates, we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable. Through experiments on MNIST and Omniglot with a variety of architectures, we show how our framework sheds light on many recent proposed extensions to the variational autoencoder family.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 - I think that VAEs are rather forced to be interpreted from an information theoretic point of view for the sake of it, rather than for the sake of a clear and unequivocal contribution from the perspective of VAEs and latent-variable models themselves. How is that useful for a VAE? - "The left vertical line corresponds to the zero rate setting. ...": All these limits are again from an information theoretic point of view and no formulation nor demonstration is provided on how this can actually be as useful. As mentioned earlier in the paper, there are well-known problems with taking this information theory perspective, e.g. difficulties in estimating MI values, etc. - Breaking (some of) the long sentences and paragraphs in page 3 with an unequivocal mathematical formulation would smooth the flow a bit. - "(2) an upper bound that measures the rate, or how costly it is to transmit information about the latent variable.": I am not entirely sure about this one and why it is massively important to be compromised against the obviously big first term. - Toy Model experiment: I do not see any indication of how this is not just a lucky catch and that VAEs consistently suffer from a problem leading to such effect. - Section 5: "can shed light on many different models and objectives that have been proposed in the literature... ": Again the contribution aspect is not so clear through the word "shed light". Minor - Although apparently VAEs represent the main and most influential latent-variable model example, I think switching too much between citing them as VAEs and then as latent-variable models in general was a bit confusing. I propose mentioning in the beginning (as happened) that VAEs are the seminal example of latent-variable models and then going on from this point onwards with VAEs without too much alternation between latent-variable models and VAEs. - page 8: "as show"
assistant
 - I think that VAEs are rather forced to be interpreted from an information theoretic point of view for the sake of it, rather than for the sake of a clear and unequivocal contribution from the perspective of VAEs and latent-variable models themselves. How is that useful for a VAE? - "The left vertical line corresponds to the zero rate setting. ...": All these limits are again from an information theoretic point of view and no formulation nor demonstration is provided on how this can actually be as useful. As mentioned earlier in the paper, there are well-known problems with taking this information theory perspective, e.g. difficulties in estimating MI values, etc. - Breaking (some of) the long sentences and paragraphs in page 3 with an unequivocal mathematical formulation would smooth the flow a bit. - "(2) an upper bound that measures the rate, or how costly it is to transmit information about the latent variable. ": I am not entirely sure about this one and why it is massively important to be compromised against the obviously big first term. - Toy Model experiment: I do not see any indication of how this is not just a lucky catch and that VAEs consistently suffer from a problem leading to such effect. - Section 5: "can shed light on many different models and objectives that have been proposed in the literature... ": Again the contribution aspect is not so clear through the word "shed light". Minor - Although apparently VAEs represent the main and most influential latent-variable model example, I think switching too much between citing them as VAEs and then as latent-variable models in general was a bit confusing. I propose mentioning in the beginning (as happened) that VAEs are the seminal example of latent-variable models and then going on from this point onwards with VAEs without too much alternation between latent-variable models and VAEs. - page 8: "as show"