A Neural Compositional Paradigm for Image Captioning
Mainstream captioning models often follow a sequential structure to generate captions, leading to issues such as introduction of irrelevant semantics, lack of diversity in the generated captions, and inadequate generalization performance. In this paper, we present an alternative paradigm for image captioning, which factorizes the captioning procedure into two stages: (1) extracting an explicit semantic representation from the given image; and (2) constructing the caption based on a recursive compositional procedure in a bottom-up manner. Compared to conventional ones, our paradigm better preserves the semantic content through an explicit factorization of semantics and syntax. By using the compositional generation procedure, caption construction follows a recursive structure, which naturally fits the properties of human language. Moreover, the proposed compositional procedure requires less data to train, generalizes better, and yields more diverse captions.
*Summary* The paper proposes a novel approach to image captioning, which adopts a recursive approach to assembling concepts detected from a visual pipeline, based on picking noun phrases and connecting them with relation phrases, which disentangles the content from the syntacic information (to a first order). The paper then shows that using such an explicit factorization in the generation mecahnism can help achieve better diversity in captions, and better generalization with lesser amount of data, and really good results on the SPICE metric which focusses more on the syntactic structure of image cpations.  *Strengths* + The paper has a pretty comprehensive list of references + The experimental results seem really thorough and consistently done, compared to most image captioning papers. The same CNN is used for all the methods and hyperparameters are all set very systematically. + It is very interesting that the proposed approach does really well on SPICE, but significantly poorly on other metrics, and it makes a lot of sense given the SPICE metric that such a method should do well on it, as well. + The experiment on using the ground truth noun phrases and using the compositional part of the model is also really nice, and gives an insight into the what the model has learnt and how performance with this paradigm might improve with better visual detectors.  *Weakness*  1. Related work which would be good to cite is [A], which also first predicts the noun phrases and then uses that to influence the structure of the captionr, and It would be good to compare and contrast both conceptually what the differences are and otherwise. 2. L133-134: Not clear what we mean by not only the complete captions but also parts thereof, and generally argument around why filling in the blanks is hard is unconvincing here. Note that there are ways to do bi-directional beam search for fill in the blanks kind of applications [C]. 3. How time consuming is the forward pass to create the caption? It seems like a pretty computationally intensive approach. How sensitive is performance to the number of noun phrases which get selected? (L202). 4. How do the statistics of the relation phrases look like? How often are the rarer relation phrases actually predicted in a sentence? Chunking things into relation phrases could have the disadvantage that one might see individual words but not the phrase at once and now we are stuck with the problem of heavy tail in relations (which is more severe than heavy tail in words).  5. Fig. 4: Why is there a bump between 0.0 and 0.1 for CompCap, and why is performance so high even at 0.0? It is realy confusing.  6. What is the training objective used for training the whole model? How are modules and losses set up? Are all the operations performed differentiable? It seems like the model makes hard decisions when picking which noun phrase to use or which relation phrase to use, does this mean there are only sequence level rewards and the whole model is trained via REINFORCE/ policy gradient? It seems like the paper is incomplete in this regard. (*) 7. L227: in a lot of ways the experiment in this paragraph is not fair, since the baseline approaches are learning to caption from scratch while the proposed approach is using noun phrase detectors trained on the full data. It would be good to report what happens if we also train the noun phrase detectors only on a part of the data. (*)  *Preliminary Evaluation* There is a major detail about the training objective and algorithms missing from the paper, however, the crux of the approach that is presented is very interesting, and the results as well as experimental analysis look great and are very thorough and creative. This would be an interesting contribution to the conference, but important details about the paper are missing, which should be fixed, marked with (*) above.  References: [A]: Gan, Zhe, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2016. “Semantic Compositional Networks for Visual Captioning.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1611.08002. [B]: Huang, Qiuyuan, Paul Smolensky, Xiaodong He, Li Deng, and Dapeng Wu. 2017. “Tensor Product Generation Networks.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1709.09118. [C]: Sun, Qing, Stefan Lee, and Dhruv Batra. 2017. “Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1705.08759.