Hierarchical Bayes Autoencoders
Autoencoders are powerful generative models for complex data, such as images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible semantic variations in the space of images. To address this problem, we present a new probabilistic generative model called the emph{Hierarchical Bayes Autoencoder (HBAE)}. The HBAE contains a multimodal decoder in the form of an energy-based model (EBM), instead of the commonly adopted unimodal Gaussian distribution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where a conditional generator is trained to match the EBM distribution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the conditional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for a set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and generates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able to generate complex image sets.
In this work a novel probabilistic generative model is introduced mixing several existing frameworks: The Hierarchical Bayes Autoencoder (HBAE) can be interpreted as a cGAN with a VAE encoder. They also incorporate a flexible learned multimodal decoder in the form of an EBM. The authors claim to produce stochastic reconstructions varying around the local data manifold of examples, and diverse unconditional samples. In addition, they present an extension of their HBAE formulation in order to model sets of inputs. This is one of the main contributions of this work since generative modeling of sets is a challenging and unsolved task. Although there is a clear explanation of the contributions of this paper, the motivation of this study is not precisely described. The introduction provides a good understanding of the topic; however, the authors may wish to provide several examples on the interest of the study. They provide a good description of the existing generative modeling architectures and the applications involved but not mention the importance of their contributions to the real world. The proposed formulation seems encouraging thanks to the incorporation of multimodal decoders. The derivation of HBAE is theoretically well justified, as a result, one could replicate or further work in this paper. The authors explain in detail the procedure they took in order to arrive to the final HBAE formulation giving a clear understanding of the topic they present. By describing the related works, they give an understandable perspective about the different drawbacks and differences of the existing methods. One could deduce the motivations of this work thanks to this section, however they should have been state clearer at the beginning. The authors explain in detail all the different parts of the architecture, presenting precise information about the different layers. At first sight, it seems like one could reproduce the methodology used. It is clearly explained and discussed. I think the experiments conducted to evaluate this work are not enough. The authors performed several experiments in order to prove the efficiency of their methodology by showing the generated images and investigate the qualitative results. If we look at the results in figure 4, we can observe some artifacts and blur regions in the image. Moreover, the variations on the faces sometimes seem more deformations than a different human feature. It would have been interesting to evaluate the identity preservation to further investigate the quality of the images. Although the methodology, conceptually, suggested an interesting approach, the results are not so encouraging given the quality of the images achieved by other methodologies. Apart from comparing quantitatively the capacity of the generated images with other related works, it would have been interesting to show that qualitatively by showing images from the mention methods. In general, there is some information missing in order to replicate the results and more details would have been appreciated. Regarding the experiments about the set of images as input, the quality of the images is similar to the simple HBAE. Since this is a more challenging approach and one could not compare with so many existing methodologies, it seems like they are going to the right direction to achieve the desired images. However, looking at the exposed results, the quality of the images is not realistic and has a lot of artifacts. Some other experiments could have been performed in order to show more interesting results. Questions: - Could you better explain the meaning of figure 1? I think it was a good idea to exemplify the differences but is not clear enough. - Did you perform any face alignment when dealing with the different faces databases? - Did you think about the possibility of giving a concrete condition for varying the image? - Which could be some concrete applications for your work?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Hierarchical Bayes Autoencoders
Autoencoders are powerful generative models for complex data, such as images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible semantic variations in the space of images. To address this problem, we present a new probabilistic generative model called the emph{Hierarchical Bayes Autoencoder (HBAE)}. The HBAE contains a multimodal decoder in the form of an energy-based model (EBM), instead of the commonly adopted unimodal Gaussian distribution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where a conditional generator is trained to match the EBM distribution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the conditional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for a set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and generates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able to generate complex image sets.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



In this work a novel probabilistic generative model is introduced mixing several existing frameworks: The Hierarchical Bayes Autoencoder (HBAE) can be interpreted as a cGAN with a VAE encoder. They also incorporate a flexible learned multimodal decoder in the form of an EBM. The authors claim to produce stochastic reconstructions varying around the local data manifold of examples, and diverse unconditional samples. In addition, they present an extension of their HBAE formulation in order to model sets of inputs. This is one of the main contributions of this work since generative modeling of sets is a challenging and unsolved task. Although there is a clear explanation of the contributions of this paper, the motivation of this study is not precisely described. The introduction provides a good understanding of the topic; however, the authors may wish to provide several examples on the interest of the study. They provide a good description of the existing generative modeling architectures and the applications involved but not mention the importance of their contributions to the real world. The proposed formulation seems encouraging thanks to the incorporation of multimodal decoders. The derivation of HBAE is theoretically well justified, as a result, one could replicate or further work in this paper. The authors explain in detail the procedure they took in order to arrive to the final HBAE formulation giving a clear understanding of the topic they present. By describing the related works, they give an understandable perspective about the different drawbacks and differences of the existing methods. One could deduce the motivations of this work thanks to this section, however they should have been state clearer at the beginning. The authors explain in detail all the different parts of the architecture, presenting precise information about the different layers. At first sight, it seems like one could reproduce the methodology used. It is clearly explained and discussed. I think the experiments conducted to evaluate this work are not enough. The authors performed several experiments in order to prove the efficiency of their methodology by showing the generated images and investigate the qualitative results. If we look at the results in figure 4, we can observe some artifacts and blur regions in the image. Moreover, the variations on the faces sometimes seem more deformations than a different human feature. It would have been interesting to evaluate the identity preservation to further investigate the quality of the images. Although the methodology, conceptually, suggested an interesting approach, the results are not so encouraging given the quality of the images achieved by other methodologies. Apart from comparing quantitatively the capacity of the generated images with other related works, it would have been interesting to show that qualitatively by showing images from the mention methods. In general, there is some information missing in order to replicate the results and more details would have been appreciated. Regarding the experiments about the set of images as input, the quality of the images is similar to the simple HBAE. Since this is a more challenging approach and one could not compare with so many existing methodologies, it seems like they are going to the right direction to achieve the desired images. However, looking at the exposed results, the quality of the images is not realistic and has a lot of artifacts. Some other experiments could have been performed in order to show more interesting results. Questions: - Could you better explain the meaning of figure 1? I think it was a good idea to exemplify the differences but is not clear enough. - Did you perform any face alignment when dealing with the different faces databases? - Did you think about the possibility of giving a concrete condition for varying the image? - Which could be some concrete applications for your work?
assistant
In this work a novel probabilistic generative model is introduced mixing several existing frameworks: The Hierarchical Bayes Autoencoder (HBAE) can be interpreted as a cGAN with a VAE encoder. They also incorporate a flexible learned multimodal decoder in the form of an EBM. The authors claim to produce stochastic reconstructions varying around the local data manifold of examples, and diverse unconditional samples. In addition, they present an extension of their HBAE formulation in order to model sets of inputs. This is one of the main contributions of this work since generative modeling of sets is a challenging and unsolved task. Although there is a clear explanation of the contributions of this paper, the motivation of this study is not precisely described. The introduction provides a good understanding of the topic; however, the authors may wish to provide several examples on the interest of the study. They provide a good description of the existing generative modeling architectures and the applications involved but not mention the importance of their contributions to the real world. The proposed formulation seems encouraging thanks to the incorporation of multimodal decoders. The derivation of HBAE is theoretically well justified, as a result, one could replicate or further work in this paper. The authors explain in detail the procedure they took in order to arrive to the final HBAE formulation giving a clear understanding of the topic they present. By describing the related works, they give an understandable perspective about the different drawbacks and differences of the existing methods. One could deduce the motivations of this work thanks to this section, however they should have been state clearer at the beginning. The authors explain in detail all the different parts of the architecture, presenting precise information about the different layers. At first sight, it seems like one could reproduce the methodology used. It is clearly explained and discussed. I think the experiments conducted to evaluate this work are not enough. The authors performed several experiments in order to prove the efficiency of their methodology by showing the generated images and investigate the qualitative results. If we look at the results in figure 4, we can observe some artifacts and blur regions in the image. Moreover, the variations on the faces sometimes seem more deformations than a different human feature. It would have been interesting to evaluate the identity preservation to further investigate the quality of the images. Although the methodology, conceptually, suggested an interesting approach, the results are not so encouraging given the quality of the images achieved by other methodologies. Apart from comparing quantitatively the capacity of the generated images with other related works, it would have been interesting to show that qualitatively by showing images from the mention methods. In general, there is some information missing in order to replicate the results and more details would have been appreciated. Regarding the experiments about the set of images as input, the quality of the images is similar to the simple HBAE. Since this is a more challenging approach and one could not compare with so many existing methodologies, it seems like they are going to the right direction to achieve the desired images. However, looking at the exposed results, the quality of the images is not realistic and has a lot of artifacts. Some other experiments could have been performed in order to show more interesting results. Questions: - Could you better explain the meaning of figure 1? I think it was a good idea to exemplify the differences but is not clear enough. - Did you perform any face alignment when dealing with the different faces databases? - Did you think about the possibility of giving a concrete condition for varying the image? - Which could be some concrete applications for your work?