Teaching Machines to Describe Images via Natural Language Feedback
Robots will eventually be part of every household. It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. We argue that a descriptive sentence can provide a much stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the problem of image captioning in which the quality of the output can easily be judged by non-experts. In particular, we first train a captioning model on a subset of images paired with human written captions. We then let the model describe new images and collect human feedback on the generated descriptions. We propose a hierarchical phrase-based captioning model, and design a feedback network that provides reward to the learner by conditioning on the human-provided feedback. We show that by exploiting descriptive feedback on new images our model learns to perform better than when given human written captions on these images.
The paper presents an approach for automatically captioning images where the model also incorporates natural language feedback from humans along with ground truth captions during training. The proposed approach uses reinforcement learning to train a phrase based captioning model where the model is first trained using maximum likelihood training (supervised learning) and then further finetuned using reinforcement learning where the reward is weighted sum of BLEU scores w.r.t to the ground truth and the feedback sentences provided by humans. The reward also consists of phrase level rewards obtained by using the human feedback.
The proposed model is trained and evaluated on MSCOCO image caption data. The proposed model is compared with a pure supervised learning (SL) model, a model trained using reinforcement learning (RL) without any feedback. The proposed model outperforms the pure SL model by a large margin and the RL model by a small margin.
Strengths:
1.	The paper is well motivated with the idea of using human in the loop for training image captioning models.
2.	The baselines (SL and RL) are reasonable and the additional experiment of using 1 GT vs. 1 feedback caption is insightful and interesting.
3.	The work can be great significance especially if the improvements are significantly large over the RL without any feedback baseline.
Weaknesses:
1.	The paper is motivated with using natural language feedback just as humans would provide while teaching a child. However, in addition to natural language feedback, the proposed feedback network also uses three additional pieces of information – which phrase is incorrect, what is the correct phrase, and what is the type of the mistake. Using these additional pieces is more than just natural language feedback. So I would like the authors to be clearer about this in introduction.
2.	The improvements of the proposed model over the RL without feedback model is not so high (row3 vs. row4 in table 6), in fact a bit worse for BLEU-1. So, I would like the authors to verify if the improvements are statistically significant.
3.	How much does the information about incorrect phrase / corrected phrase and the information about the type of the mistake help the feedback network? What is the performance without each of these two types of information and what is the performance with just the natural language feedback?
4.	In figure 1 caption, the paper mentions that in training the feedback network, along with the natural language feedback sentence, the phrase marked as incorrect by the annotator and the corrected phrase is also used. However, from equations 1-4, it is not clear where the information about incorrect phrase and corrected phrase is used. Also L175 and L176 are not clear. What do the authors mean by “as an example”?  
5.	L216-217: What is the rationale behind using cross entropy for first (P – floor(t/m)) phrases? How is the performance when using reinforcement algorithm for all phrases?
6.	L222: Why is the official test set of MSCOCO not used for reporting results?
7.	FBN results (table 5): can authors please throw light on why the performance degrades when using the additional information about missing/wrong/redundant?
8.	Table 6: can authors please clarify why the MLEC accuracy using ROUGE-L is so low? Is that a typo?
9.	Can authors discuss the failure cases of the proposed (RLF) network in order to guide future research?
10.	Other errors/typos:
a.	L190: complete -> completed
b.	L201, “We use either … feedback collection”: incorrect phrasing
c.	L218: multiply -> multiple
d.	L235: drop “by”
Post-rebuttal comments:
I agree that proper evaluation is critical. Hence I would like the authors to verify that the baseline results [33] are comparable and the proposed model is adding on top of that.
So, I would like to change my rating to marginally below acceptance threshold.