When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, or if the model hallucinates based on priors in the dataset and/or the language model. The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word. The model must therefore learn to predict the attentional weights without knowing the word it should localize. This is difficult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words. In this work, we help the model to achieve this via a novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth. Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time. We show that our model significantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference for both image and video captioning tasks.
# 1. Summary
The paper deals with the problem of learning grounded captions from images without joint text-location information, but instead texts (words in captions) and locations are provided independently and the model needs to figure out their link. The model is built upon GVD (Zhou et al., 2019): each word generated by the encoder is grounded to the locations provided by the region proposal module (Up-Down model (Anderson et al., 2018)), used to reconstruct the ground-truth caption.     
My weak reject decision was guided by the following strengths and weaknesses of the paper.
Strengths:
* The reconstruction formulation of the problem is interesting and relevant for the task
* Proposed new metric to measure grounding performance
      
Weaknesses:
* Questionable motivations: it is not clear what application is grounding text to the image useful for?
* Marginal (not statistically significant?) improvement on image captioning metrics by using grounded text (proposed method) compared to not grounding (GVD)
* Limited novelty: extension of GVD, where attention is removed and object locations are used instead
     
      
# 2. Clarity and Motivation
The paper is generally well written, however there are some concerns on motivations. 
One concern is related to the motivations of the paper. The authors use grounding as a proxy to improve image captioning results, which improvement is marginal wrt GVD (see Table 1). Why do we need to localize text if this has very marginal impact on the captioning metrics? It is missing the link between the potential applications where the localization of words is relevant. 
The authors claim that they do not uses any grounding annotation, however the pre-trained Faster-RCNN has been trained using annotations which consist of bounding boxes + categories. Therefore, the model do (in an implicit way) rely on grounding annotations, especially because there might be an overlap between the words (classes) used to pretrain the detector and the words in the captions. The authors should assess if this ovelap/bias in the pre-trained Faster-RCNN exists or not.
Some other questions are still to be answered:
* How are the regions R parametrized? Is it the visual representation or bounding box locations?
* What happens to words that are not grounded to the image (e.g., verbs or articles)? Do you have a special way to deal with those?
* What is the intuition of multiplying word embedding and region embeddings to generate z in Eq. 6? 
# 3. Novelty
The proposed method is an extension of the existing model GVD, where the attentional module is removed and its functionality is replaced by the cyclical training mode with reconstruction of the object locations. From the technical point of view this is limited novelty, but still an interesting improvement of the model; however the experiments and results do not support the claim that using such model improves image captioning result in a significant way. One way to answer to this question would have been by showing an application where the outputted locations are used for downstream tasks.
# 4. Experimentation
The experiments are carried out in a scrupulous way, by showing the comparing with GVD (with and without attention; with and without grounding supervision). The non-convincing part of them (as mentioned above already) is the fact that the improvements on these datasets might be non significant for image captioning. For example, let's consider the image captioning results in Table 1 (Flickr30k Entities): cyclical have a max improvement of 0.7 (CIDER) and min of 0 (B@4) when compared with GVD without grounding supervision. There is an obvious huge improvement on the grounding evaluation, which is obvious since GVD does not do it explicitly. The same trend is in Table 2. 
These results are not convincing, combined by the fact that it is not clear in which applications one would want a very accurate grounded text.
# Minor Points
* Sec. 3.1: it is not clear that the Language LSTM is the decoder. Please explicitly say it before Eq. 1
* Caption of Table 3: which dataset is this?