Published as a conference paper at ICLR 2018 EMERGENT TRANSLATION IN MULTI-AGENT COMMUNICATION
While most machine translation systems to date are trained on large parallel corpora, humans learn language in a different way: by being grounded in an environment and interacting with other humans. In this work, we propose a communication game where two agents, native speakers of their own respective languages, jointly learn to solve a visual referential task. We find that the ability to understand and translate a foreign language emerges as a means to achieve shared goals. The emergent translation is interactive and multimodal, and crucially does not require parallel corpora, but only monolingual, independent text and corresponding images. Our proposed translation model achieves this by grounding the source and target languages into a shared visual modality, and outperforms several baselines on both word-level and sentence-level translation tasks. Furthermore, we show that agents in a multilingual community learn to translate better and faster than in a bilingual communication setting.
Summary: The authors show that using visual modality as a pivot they can train a model to translate from L1 to L2. 
Please find my detailed comments/questions/suggestions below:
1) IMO, the paper could have been written much better. At the core, this is simply a model which uses images as a pivot for learning to translate between L1 and L2 by learning a common representation space for {L1, image} or {L2, image}. There are several works on such multimodal representation learning but the authors present their work in a way which makes it look very different from these works. IMO, this leads to unnecessary confusion and does more harm than good. For example, the abstract gives an impression that the authors have designed a game to collect data (and it took me a while to set this confusion aside).
2) Continuing on the above point, this is essentially about learning a common multimodal representation and then decode from this common representation. However, the authors do not cite enough work on such multimodal representation learning (for example, look at Spandana et. al.: Image Pivoting for Learning Multilingual Multimodal Representations, EMNLP 2017 for a good set of references)
3) This omission of related work also weakens the experimental section. At least for the word translation task many of these common representation learning frameworks could have been easily evaluated. For example, find the nearest german neighbour of the word "dog" in the common representation space. The authors instead compare with very simple baselines.
4) Even when comparing with simple baselines, the proposed model does not convincingly outperform them. In particular,  the P@5 and P@20 numbers are only slightly better. 
5) Some of the choices made in the Experimental setup seem questionable to me:
   - Why  use a NMT model without attention? That is not standard and does not make sense to use when a better baseline model (with attention) is available ?
   - It is mentioned that "While their model unit-normalizes the output of every encoder, we found this to consistently hurt performance, so do not use normalization for fair comparison with our models." I don't think this is a fair comparison. The authors can mention their results without normalization if that works well for them but it is not fair to drop normalization from the model of N&N if that gives better performance. Please mention the numbers with unit normalization to give a better picture. It does not make sense to weaken an existing baseline and then compare with it.
6) It would be good to mention the results of the NMT model in Table 1 itself instead of mentioning them separately in a paragraph. This again leads to poor readability and it is hard to read and compare the corresponding numbers from Table 1.  I am not sure why this cannot be accommodated in the Table itself.
7) In Figure 2, what exactly do you mean by "Results are averaged over 30 translation scenarios". Can you please elaborate ?