GENERATIVE MODELS OF VISUALLY GROUNDED IMAGINATION
It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C's). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et al. (2017) and the BiVCCA method of Wang et al. (2016b)) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset (Liu et al., 2015).
This paper presented a multi-modal extension of variational autoencoder (VAE) for the task "visually grounded imagination."  In this task,  the model learns a joint embedding of the images and the attributes. The proposed model is novel but incremental comparing to existing frameworks.  The author also introduced new evaluation metrics to evaluate the model performance concerning correctness, coverage, and compositionality. 
Pros:
1. The paper is well-written, and the contribution (both the model and the evaluation metric) potentially can to be very useful in the community.  
2. The discussion comparing the related work/baseline methods is insightful. 
3. The proposed model addresses many important problems, such as attribute learning, disentanged representation learning, learning with missing values, and proper evaluation methods. 
Cons/questions:
1. The motivation of the model choice of q is not clear.
Comparing to BiVCCA, apart from the differences that the author discussed, a big difference is the choice of q.  BiVCCA uses two inference networks q(z|x) and q(z|y), while the proposed method uses three. q(z|x), q(z|y), and q(z|x,y).  How does such model choice affect the final performance? 
2. Baselines are not necessarily sufficient. 
The paper compared the vanilla version of BiVCCA but not the one with factorized representation version. In the original VAECCA paper, the extension of using factorized representation (private and shared) improved the performance]. The author should also compare this extension of VAECCA.
3. Some details are not clear. 
a) How to set/learn the scaling parameter \lambda_y and \beta_y? If it is set as hyper-parameter, how does the performance change concerning them? 
b) Discussion of the experimental results is not sufficient. For example, why JMVAE performs much better than the proposed model when all attributes are given.  What is the conclusion from Figure 4(b)? The JMVAE seems to generate more diverse (better coverage) results which are not consistent with the claims in the related work.  The same applies to figure 5.