Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straightthrough Gumbel-softmax estimator (Jang et al., 2017)) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.
Increasing my score based on the authors rebuttal. The argument that the proposed method can complement human-bot training makes sense. Also, it seems RL baseline experiments were exhaustive. But the argument about the learnt language being compositional should be toned down since there is not enough evidence to support it.
Old reviews:
The paper proposes to use Gumbel-softmax for training sender and receiver agents in a referential game like Lazaridou (2016). Unlike the previous work, large number of distraction images and variable length messages are allowed. The results suggest the proposed method outperforms Reinforce baseline. In addition, the paper proposes two way to force the communication to be more like natural language.
My first concern is that the paper disregards continuous communication for being unlike natural language, and insists on using discrete symbols. But during training, continuous communication is allowed transmit gradients between the agents. This use of back-propagation makes it impossible to learn from interactions with human, which is probably vital for learning natural language. 
Secondly, I think baseline experiments are not exhaustive and can be improved. Here are my points:
- L109: Learnt baseline (network itself predicts baseline values) is pretty standard in deep RL works for it is efficiency over moving average. Why it hasn't tried? Also it is common to add entropy regularization term to the loss to encourage exploration in deep RL. Has it tried? 
- Appropriate baseline here continuous communication during training and discretization during testing as done in (Foerster, 2016) 
- L135: a simple solution would be to decrease the temperature to zero during training. Has it tried?
- L163: why not hyper-parameters are not tuned? At least the learning rate should be tuned, especially in Reinforce baseline because the default values are tuned for supervised training.
At last, the claim of compositionally (L11) of the learnt communication is not supported by any evidence. Also, L196 suggests that word 5747 represents animals. Then, it should still represent the same thing even at different position as in natural language word "animal". The fact that it is not suggest that the learnt communication is actually unlike natural language. Why not similar analysis on reinforce baseline?
Other comments:
- L59: It is not a non-stationary environment. Since the sender and the receiver have the same reward, they can be considered as a single RL agent. Then, this agent writes words into the environment as "sender", then observes them at the next step as "receiver". 
- How KL is optimized in sec 3.3? It can't be computed directly because there are exponentially many "m"s.
- Why there are two reinforce versions in figure 2 left. What is greedy argmax reinforce?
- It would be good to include a histogram of message lengths
- Why such large vocabulary size? What is the effect of smaller vocabulary?