Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
This paper explores image caption generation using conditional variational autoencoders (CVAEs). Standard CVAEs with a fixed Gaussian prior yield descriptions with too little variability. Instead, we propose two models that explicitly structure the latent space around K components corresponding to different types of image content, and combine components to create priors for images that contain multiple types of content simultaneously (e.g., several kinds of objects). Our first model uses a Gaussian Mixture model (GMM) prior, while the second one defines a novel Additive Gaussian (AG) prior that linearly combines component means. We show that both models produce captions that are more diverse and more accurate than a strong LSTM baseline or a "vanilla" CVAE with a fixed Gaussian prior, with AG-CVAE showing particular promise.
This paper investigated the task of image-conditioned caption generation using deep generative models. Compared to existing methods with pure LSTM pipeline, the proposed approach augments the representation with an additional data dependent latent variable. This paper formulated the problem under variational auto-encoder (VAE) framework by maximizing the variational lowerbound as objective during training. A data-dependent additive Gaussian prior was introduced to address the issue of limited representation power when applying VAEs to caption generation. Empirical results demonstrate the proposed method is able to generate diverse and accurate sentences compared to pure LSTM baseline. 
== Qualitative Assessment ==
I like the motivation that adding stochastic latent variable to the caption generation framework. Augmenting the prior of VAE is not a novel contribution but I see the novelty applied to caption generation task. Performance-wise, the proposed AG-CVAE achieved more accurate performance compared to both LSTM baseline and other CVAE baselines (see Table 2). The paper also analyzed the diversity of the generated captions in comparison with the pure LSTM based approach (see Figure 5). Overall, the paper is generally well-written with sufficient experimental details.
Considering the additive Gaussian prior as major contribution, the current version does not seem to be very convincing to me. I am happy to raise score if my concerns can be addressed in the rebuttal.
* Any strong evidence showing the advantages of AG-CVAE over CVAE/GMM-CVAE?
The improvements in Table 2 are not very significant. For qualitative results (Figure 5 and other figures in supplementary materials), side-by-side comparisons between AG-CVAE and CVAE/GMM-CVAE are missing. It is not crystal clear to me whether AG-CVAE actually adds more representation power compared to CVAE/GMM-CVAE. Please comment on this in the rebuttal and include such comparisons in the final version of the paper (or supplementary materials).
* Diversity evaluation: it is not clear why does AG-CVAE perform worse than CVAE. Also, there is no explanation about performance gap from different variations of AG-CVAE. Since CVAE is a stochastic generative model, I wonder whether top 10 sentences are sufficient for diversity evaluations? The results will be much more convincing if the authors provide a curve (y-axis is the diversity measure and x-axis is the number of sentences).
Additional comments:
* Pre-defined means of clusters for GMM-CVAE and AG-CVAE (Line 183)
It is not surprising that the authors failed to obtain better results when means of clusters u_k are free variables. In principle, it is possible to learn duplicate representations without any constraints (e.g., sparsity or orthogonality) on u_k or c_k. I would encourage the authors to explore this direction a bit in the future. Hopefully, learnable data-dependent prior can boost the performance to some extent.