Inspired by previous work on emergent communication in referential games, we propose a novel multi-modal, multi-step referential game, where the sender and receiver have access to distinct modalities of an object, and their information exchange is bidirectional and of arbitrary duration. The multi-modal multi-step setting allows agents to develop an internal communication significantly closer to natural language, in that they share a single set of messages, and that the length of the conversation may vary according to the difficulty of the task. We examine these properties empirically using a dataset consisting of images and textual descriptions of mammals, where the agents are tasked with identifying the correct object. Our experiments indicate that a robust and efficient communication protocol emerges, where gradual information exchange informs better predictions and higher communication bandwidth improves generalization.
--------------
Summary and Evaluation:
--------------
The paper presents a nice set of experiments on language emergence in a mutli-modal, multi-step setting. The multi-modal reference game provides an interesting setting for communication, with agents learning to map descriptions to images. The receiving agent's direct control over dialog length is also novel and allows for the interesting analysis presented in later sections. 
Overall I think this is an interesting and well-designed work; however, some details are missing that I think would make for a stronger submission (see weaknesses).
--------------
Strengths:
--------------
- Generally well-written with the Results and Analysis section appearing especially thought-out and nicely presented.
- The proposed reference game provides a number of novel contributions -- giving the agents control over dialog length, providing both agents with the same vocabulary without constraints on how each uses it (implicit through pretraining or explicit in the structure/loss), and introducing an asymmetric multi-modal context for the dialog.
- The analysis is extensive and well-grounded in the three key hypothesis presented at the beginning of Section 6.
--------------
Weaknesses:
--------------
- There is room to improve the clarity of Sections 3 and 4 and I encourage the authors to revisit these sections. Some specific suggestions that might help:
		- numbering all display style equations
		- when describing the recurrent receiver, explain the case where it terminates (s^t=1) first such that P(o_r=1) is defined prior to being used in the message generation equation. 
- I did not see an argument in support of the accuracy@K metric. Why is putting the ground truth in the top 10% the appropriate metric in this setting? Is it to enable comparison between the in-domain, out-domain, and transfer settings?
- Unless I missed something, the transfer test set results only comes up once in the context of attention methods and are not mentioned elsewhere. Why is this? It seems appropriate to include in Figure 5 if no where else in the analysis.
- Do the authors have a sense for how sensitive these results are to different runs of the training process?
- I did not understand this line from Section 5.1: "and discarding any image with a category beyond the 398-th most frequent one, as classified by a pretrained ImageNet classifier'"
- It is not specified (or I missed it) whether the F1 scores from the separate classifier are from training or test set evaluations.
- I would have liked to see analysis on the training process such as a plot of reward (or baseline adjusted reward) over training iterations. 
- I encourage authors to see the EMNLP 2017 paper "Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog" which also perform multi-round dialogs between two agents. Like this work, the authors also proposed removing memory from one of the agents as a means to avoid learning degenerate 'non-dialog' protocols.
- Very minor point: the use of fixed-length, non-sequence style utterances is somewhat disappointing given the other steps made in the paper to make the reference game more 'human like' such as early termination, shared vocabularies, and unconstrained utterance types. I understand however that this is left as future work.
--------------
Curiosities:
--------------
- I think the analysis is Figure 3 b,c is interesting and wonder if something similar can be computed over all examples. One option would be to plot accuracy@k for different utterance indexes -- essentially forcing the model to make a prediction after each round of dialog (or simply repeating its prediction if the model has chosen to stop).