We present an extension of a variational auto-encoder that creates semantically rich coupled probabilistic latent representations that capture the semantics of multiple modalities of data. We demonstrate this model through experiments using images and textual descriptors as inputs and images as outputs. Our latent representations are not only capable of driving a decoder to generate novel data, but can also be used directly for annotation or classification. Using the MNIST and Fashion-MNIST datasets we show that the embedding not only provides better reconstruction and classification performance than the current state-of-the-art, but it also allows us to exploit the semantic content of the pretrained word embedding spaces to do tasks such as image generation from labels outside of those seen during training.
Summary
=========
The authors present an extension to the VAE model by exploring the possibility of using the label space to create a new embedding space, which they call Probabilistic Semantic Embedding (PSE). 
They present two different extension of PSE, PSE and PSE*.
The idea of additionally supporting the latent embedding, created by a VAE, by using available textual descriptors seems promising. 
The proposed model was evaluated on two tasks, label-to-image generation and image annotation.
Although the work is interesting, there are a few questions that I am not clear about and have several comments. 
Questions
=========
1. How was the word2vec model trained? Did you use an existing pretrained model (e.g. available as download) or did you train the embedding model yourself? If so, on what data? 
2. The major novelty of this approach is the use of annotations supporting images and textual (pretrained) embedding spaces, but no related work regarding Wes was neither introduced in the Related Work section nor was it clearly explained in the text.
3. Why did the authors focus on the w2v model instead of more promising approaches as fastText or ELMo? 
4. How does your model deal with OOV word(s) as input? For example, when used as Image Generator.
5. Table 1 shows results achieved on MNIST but not Fashion-MNIST; was the evaluation performed on MNIST only? 
6. Table 2 presents the impact of the use of pretrained embeddings (word2vec) instead of one-hot vectors for labels. Which one do the models presented in Table 1 use? 
7. Could you explain the small difference between using one-hot vs pretrained label encodings, presented in Table 2? 
8. Also, can you explain how the numbers in table 2 were achieved (e.g. sum over all, average of all, etc.). When comparing the values presented here to the values of the same measure in table 1, one does notice the big difference between them. 
Comments
=========
1. Section 2, page2: “As derived in the original paper…” references which paper (i.e. Kingma et al)? 
2. VAE or beta-VAE model is not referenced (mentioned on page 5);
3. Authors do agree that the corpora used is not optimal for the adequate evaluation of the proposed model. It would be interesting to see the use of this approach on a more realistic data set;
4. Unclear sentence: “Compared with the VAE, latent codes where images with the same labels are clustered.”;
5. The authors claim that one of the results of this work is the possibility to generalize for unseen cases (zero-shot learning). It would be interesting to see the performance of this model compared to SOTA in CV in terms of the zero-shot learning task;
6. Figure 2 visualizes proposed embedding space (2D) but it shows VAE and beta-VAE models and omits to show PSE. VAE and beta-VAE are neither introduced nor referenced in text;
7. Table 1: mark best performing with bold. It does, however, outperform other evaluated models when using 20D embedding space;
8. Page 7: in text you mention generation accuracy and in Table 1 the same value is defined as Generation Correctness (%).