Review Networks for Caption Generation
We propose a novel extension of the encoder-decoder framework, called a review network. The review network is generic and can enhance any existing encoderdecoder model: in this paper, we consider RNN decoders with both CNN and RNN encoders. The review network performs a number of review steps with attention mechanism on the encoder hidden states, and outputs a thought vector after each review step; the thought vectors are used as the input of the attention mechanism in the decoder. We show that conventional encoder-decoders are a special case of our framework. Empirically, we show that our framework improves over state-ofthe-art encoder-decoder systems on the tasks of image captioning and source code captioning.
The paper proposes a novel sequence-to-sequence architecture that includes an encoder and a decoder, as usual, but adds a "reviewer" module in the middle that can perform a fixed number of attentive iterations over the input representations in order to better represent the complete input sequence. This process can also be refined by adding an intermediate loss to guide it, for instance a bag-of-word loss over the target sequence. Experiments are conducted on two different tasks (image captioning and code captioning) and comparisons are made with other state-of-the-art approaches. 
In the last few months, a few related approaches have been proposed that should probably be considered. I remember (at least): - Order Matters:..., by Vinyals et al, ICLR 2016 - Adaptive Computation..., by Graves, ArXiv 2016. The "Order Matters" paper proposes a "Process" module between the encoder and the decoder which has a lot of similarities with the "Attentive Input Reviewer" version I think. The "Adaptive Computation" paper bears similarities with the Decoder module I think.  I like the idea of the "discriminative supervision" which enable to re-use the same supervision in a different way (kind of bag-of-word supervision) but I would really like to see how important it is: it seems that in the experiment the \lambda factor that mediates between this loss and the usual loss is fixed, so we don't really know how important this is.  The experiments on COCO do not mention that on the MSCOCO website, more recent results are available and show better performance on about all metrics compared to the ones in the paper.  Another interesting experiment I wish was provided was the importance of T_r, which is set to 8 in the COCO experiment. What happens when this is changed (lower? higher? is there potential overfitting?).  Regarding the image captioning model using VGG, could you specify where was the input attention taken? (which layer of VGG)  Regarding the attentive input reviewer vs attentive output reviewer, I would have liked to have a discussion on when to use which and how they compare.  ========  I have read the author's response, which answered all my concerns. I think the updated paper should be accepted.