While reinforcement learning (RL) shows a lot of promise for natural language processing-e.g. when fine-tuning natural language systems for optimizing a certain objective-there has been little investigation into potential language drift: when an external reward is used to train a system, the agents' communication protocol may easily and radically diverge from natural language. By re-casting translation as a communication game, we show that language drift indeed happens when pre-trained agents are fine-tuned with policy gradient methods. We contend that simply adding a "naturalness" constraint to the reward, e.g. by using language model log likelihood, does not fully address the issue, and argue that (perceptual) grounding is required. That is, while language model constraints impose syntactic conformity, they do not lead to semantic correspondence. Our experiments show that grounded models give the best communication performance, while retaining English syntax along with the ability to convey the intended semantics.
This paper poses and addresses the problem of language drift in multi-agent communication paradigms. When two pretrained natural-language agents are jointly optimized to communicate and solve some external non-linguistic objective, their internal communication often diverges to a code-like, unnatural communication system. This paper solves this “language drift” problem by requiring that the messages between agents be usable as inputs to an image caption retrieval system. They demonstrate that the jointly optimized agents perform best when regularized in this manner to prevent language drift.
1. Framing: I’m uncertain about the framing of this paper. The authors pose the problem of “language drift,” which is indeed a frequent problem in multi-agent communication tasks where the principle supervision involves non-linguistic inputs and outputs. They then design a three-language MT task as a test case, where the inputs and outputs are both linguistic. Why attack this particular task and grounding solution? I can imagine some potential goals of the paper, but also see more direct ways to address each of the potential goals than what the authors have chosen:
1a. Study how to counter language drift in general — why not choose a more intuitive two-agent communication task, e.g. navigation, game playing, etc.?
1b. Study how to counter language drift in the MT task — aren’t there simpler solutions to prevent language drift in this particular task? e.g. require “cycle-consistency” – that it be possible to reconstruct the French input using the French output? Why pick multimodal grounding, given that it imposes substantial additional data requirements?
1c. Build a better/more data-efficient machine translation system — this could be an interesting goal and suitable for the paper, but this doesn’t seem to be the framing that the authors intend.
2. Interpretation of first results:
2a. Thanks for including standard deviation estimates! I think it’s also important that you do some sort of significance testing on the comparison between PG+LM+G and PG+LM performance for Fr->En->De — these numbers look pretty close to me. You could run e.g. a simple sign test on examples within each corpus between the two conditions.
2b. It would also be good to know how robust your results are to hyperparameter settings (especially the entropy regularization hyperparameter).
3. Token frequency results: These are intriguing but quite confusing to me!
3a. How sensitive are these results to your entropy regularization setup? How does PG behave without entropy regularization?
3b. Table 6 shows that the PG model has very different drift for different POS categories. Does this explain away the change in the token frequency distribution? What do the token frequency effects look like for PG within the open-class / content word categories (i.e., controlling for the huge difference in closed-class behavior)?
4. Minor comments:
4a. There’s a related problem in unsupervised representation learning for language. Work on VAEs for language, for example, has shown that the encoder often collapses meaning differences in the latent representation, and leans on an overly powerful decoder in order to pick up all of the lost information. It would be good to reference this work in your framing (see e.g. Bowman et al. (2015)).
4b. In sec. 3.1 you overload notation for R. Can you subscript these so that it’s especially clear in your results which systems are following which reward function?
4c. Great to show some qualitative examples in Table 7 — can you explicitly state where these are from (dev set vs. test set?) and whether they are randomly sampled?
References:
Bowman et al. (2015). Generating sentences from a continuous space. https://arxiv.org/abs/1511.06349