Hierarchical Adversarially Learned Inference
We propose a novel hierarchical generative model with a simple Markovian structure and a corresponding inference model. Both the generative and inference model are trained using the adversarial learning paradigm. We demonstrate that the hierarchical structure supports the learning of progressively more abstract representations as well as providing semantically meaningful reconstructions with different levels of fidelity. Furthermore, we show that minimizing the Jensen-Shanon divergence between the generative and inference network is enough to minimize the reconstruction error. The resulting semantically meaningful hierarchical latent structure discovery is exemplified on the CelebA dataset. There, we show that the features learned by our model in an unsupervised way outperform the best handcrafted features. Furthermore, the extracted features remain competitive when compared to several recent deep supervised approaches on an attribute prediction task on CelebA. Finally, we leverage the model*s inference network to achieve state-of-the-art performance on a semi-supervised variant of the MNIST digit classification task.
 ****** Please note the adjusted review score after revisions and clarifications of the authors. The paper was improved significantly but still lacks novelty. For context, multi-layer VAEs also were not published unmodified as follow-up papers since the objective is identical. Also, I would suggest the authors study the modified prior with marginal statistics and other means to understand not just *that* their model performs better with the extra degree of freedom but also *how* exactly it does it. The only evaluation is sampling from z1 and z2 for reconstruction which shows that some structure is learned in z2 and the attribute classification task. However, more statistical understanding of the distributions of the extra layers/capacity of the model would be interesting. ****** The authors propose a hierarchical GAN setup, called HALI, where they can learn multiple sets of latent variables. They utilize this in a deep generative model for image generation and manage to generate good-looking images, faithful reconstructions and good inpainting results. At the heart of the technique lies the stacking of GANS and the authors claim to be proposing a novel model here. First, Emily Denton et. al proposed a stacked version of GANs in "Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks", which goes uncited here and should be discussed as it was the first work stacking GANs, even if it did so with layer-wise pretraining. Furthermore, the differences to another very similar work to that of the authors (StackGan by Huan et al) are unclear and not well motivated. And third, the authors fail to cite *Adversarial Message Passing* by Karaletsos 2016, which has first introduced joint training of generative models with structure by hierarchical GANs and generalizes the theory to a particular form of inference for structured models with GANs in the loop. This cannot be called concurrent work as it has been around for a year and has been seen and discussed at length in the community, but the authors fail to acknowledge that their basic idea of a joint generative model and inference procedure is subsumed there. In addition, the authors also do not offer any novel technical insights compared to that paper and actually fall short in positioning their paper in the broader context of approximate inference for generative models. Given these failings, this paper has very little novelty and does not perform accurate attribution of credit to the community. Also, the authors propose particular one-off models and do not generalize this technique to an inference principle that could be reusable. As to its merits, the authors manage to get a particularly simple instance of a *deep gan* working for image generation and show the empirical benefits in terms of image generation tasks. In addition, they test their method on a semi-supervised task and show good performance, but with a lack of details. In conclusion, this paper needs to flesh out its contributions on the empirical side and position its exact contributions accordingly and improve the attribution.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Hierarchical Adversarially Learned Inference
We propose a novel hierarchical generative model with a simple Markovian structure and a corresponding inference model. Both the generative and inference model are trained using the adversarial learning paradigm. We demonstrate that the hierarchical structure supports the learning of progressively more abstract representations as well as providing semantically meaningful reconstructions with different levels of fidelity. Furthermore, we show that minimizing the Jensen-Shanon divergence between the generative and inference network is enough to minimize the reconstruction error. The resulting semantically meaningful hierarchical latent structure discovery is exemplified on the CelebA dataset. There, we show that the features learned by our model in an unsupervised way outperform the best handcrafted features. Furthermore, the extracted features remain competitive when compared to several recent deep supervised approaches on an attribute prediction task on CelebA. Finally, we leverage the model*s inference network to achieve state-of-the-art performance on a semi-supervised variant of the MNIST digit classification task.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 ****** Please note the adjusted review score after revisions and clarifications of the authors. The paper was improved significantly but still lacks novelty. For context, multi-layer VAEs also were not published unmodified as follow-up papers since the objective is identical. Also, I would suggest the authors study the modified prior with marginal statistics and other means to understand not just *that* their model performs better with the extra degree of freedom but also *how* exactly it does it. The only evaluation is sampling from z1 and z2 for reconstruction which shows that some structure is learned in z2 and the attribute classification task. However, more statistical understanding of the distributions of the extra layers/capacity of the model would be interesting. ****** The authors propose a hierarchical GAN setup, called HALI, where they can learn multiple sets of latent variables. They utilize this in a deep generative model for image generation and manage to generate good-looking images, faithful reconstructions and good inpainting results. At the heart of the technique lies the stacking of GANS and the authors claim to be proposing a novel model here. First, Emily Denton et. al proposed a stacked version of GANs in "Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks", which goes uncited here and should be discussed as it was the first work stacking GANs, even if it did so with layer-wise pretraining. Furthermore, the differences to another very similar work to that of the authors (StackGan by Huan et al) are unclear and not well motivated. And third, the authors fail to cite *Adversarial Message Passing* by Karaletsos 2016, which has first introduced joint training of generative models with structure by hierarchical GANs and generalizes the theory to a particular form of inference for structured models with GANs in the loop. This cannot be called concurrent work as it has been around for a year and has been seen and discussed at length in the community, but the authors fail to acknowledge that their basic idea of a joint generative model and inference procedure is subsumed there. In addition, the authors also do not offer any novel technical insights compared to that paper and actually fall short in positioning their paper in the broader context of approximate inference for generative models. Given these failings, this paper has very little novelty and does not perform accurate attribution of credit to the community. Also, the authors propose particular one-off models and do not generalize this technique to an inference principle that could be reusable. As to its merits, the authors manage to get a particularly simple instance of a *deep gan* working for image generation and show the empirical benefits in terms of image generation tasks. In addition, they test their method on a semi-supervised task and show good performance, but with a lack of details. In conclusion, this paper needs to flesh out its contributions on the empirical side and position its exact contributions accordingly and improve the attribution.
assistant
 ****** Please note the adjusted review score after revisions and clarifications of the authors. The paper was improved significantly but still lacks novelty. For context, multi-layer VAEs also were not published unmodified as follow-up papers since the objective is identical. Also, I would suggest the authors study the modified prior with marginal statistics and other means to understand not just *that* their model performs better with the extra degree of freedom but also *how* exactly it does it. The only evaluation is sampling from z1 and z2 for reconstruction which shows that some structure is learned in z2 and the attribute classification task. However, more statistical understanding of the distributions of the extra layers/capacity of the model would be interesting. ****** The authors propose a hierarchical GAN setup, called HALI, where they can learn multiple sets of latent variables. They utilize this in a deep generative model for image generation and manage to generate good-looking images, faithful reconstructions and good inpainting results. At the heart of the technique lies the stacking of GANS and the authors claim to be proposing a novel model here. First, Emily Denton et. al proposed a stacked version of GANs in "Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks", which goes uncited here and should be discussed as it was the first work stacking GANs, even if it did so with layer-wise pretraining. Furthermore, the differences to another very similar work to that of the authors (StackGan by Huan et al) are unclear and not well motivated. And third, the authors fail to cite *Adversarial Message Passing* by Karaletsos 2016, which has first introduced joint training of generative models with structure by hierarchical GANs and generalizes the theory to a particular form of inference for structured models with GANs in the loop. This cannot be called concurrent work as it has been around for a year and has been seen and discussed at length in the community, but the authors fail to acknowledge that their basic idea of a joint generative model and inference procedure is subsumed there. In addition, the authors also do not offer any novel technical insights compared to that paper and actually fall short in positioning their paper in the broader context of approximate inference for generative models. Given these failings, this paper has very little novelty and does not perform accurate attribution of credit to the community. Also, the authors propose particular one-off models and do not generalize this technique to an inference principle that could be reusable. As to its merits, the authors manage to get a particularly simple instance of a *deep gan* working for image generation and show the empirical benefits in terms of image generation tasks. In addition, they test their method on a semi-supervised task and show good performance, but with a lack of details. In conclusion, this paper needs to flesh out its contributions on the empirical side and position its exact contributions accordingly and improve the attribution.