A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training. However, recent work suggests that current machine learning methods are too data inefficient to be trained in this way from scratch. In this paper, we investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency: imitating human language data via supervised learning, and maximizing reward in a simulated multi-agent environment via self-play (as done in emergent communication), and introduce the term supervised self-play (S2P) for algorithms using both of these signals. We find that first training agents via supervised learning on human data followed by self-play outperforms the converse, suggesting that it is not beneficial to emerge languages from scratch. We then empirically investigate various S2P schedules that begin with supervised learning in two environments: a Lewis signaling game with symbolic inputs, and an image-based referential game with natural language descriptions. Lastly, we introduce population based approaches to S2P, which further improves the performance over single-agent methods.
Summary
---
(motivation)
To develop language speaking agents we can teach them to mimic human language
or to solve tasks that require communication. The latter is efficient, but
the former enables interpretability. Thus we combine the two in an attempt
to take advantage of both advantages. This paper studies a variety of ways to
combine these approaches to inform future work that needs to make this tradeoff.
(approach)
The trade-off is studied using reference games between a speaker and a
listener. Goal oriented _self-play_ and human _supervision_ are considered two contraints one
can put on a network during learning. This work considers algorithms that vary
when self-play and supervision are used (e.g., training with self-play then supervision,
or supervision then self-play, or alternating back and forth between the two).
Additional variations freeze the speaker or distill an ensemble of agents into one agent.
(experiments)
A synthetic Object Reference game (OR) and a Image-Base Reference game (IBR) with real images are used for evaluation. Performance is accuracy at image/object guessing.
1. (OR) Like previous work, this work finds that emergent languages are imperfect at supporting their goals and cannot be understood by agents that only understand a human language like English.
2. (OR) Pre-training with supervision then fine-tuning with self-play is superior to pre-training with self-play then fine-tuning with supervision. This is presented as surprising from the perspective of language emergence literature, which is though of as pre-training with self-play.
3. (IBR) Distilling an agent from an ensemble of 50 independently trained agents outperforms training single agents from scratch, but is still not as good as the whole ensemble.
Self-play vs supervision schedules:
4. (IBR) Supervision (using image captions) followed by self-play performs much worse than all other approaches.
5. (IBR) Alternating between supervision and self play (e.g., randomly choosing supervision or self-play every iteration) performs best.
Strengths
---
The curricula considered by this paper seem to have a sigificant impact on performance. These are new and could be important for future work on language learning, which may have considered the sup2sp setting from figure 7a without considering the sched setting.
The diversity of experiments provided and the analysis help the reader get a better sense for how emergent communication models work.
It's nice to see experiments on both a toy setting and a setting with realistic images.
Future directions suggested throughout the paper are interesting.
Weaknesses
---
* The 3rd point of section 5 is presented as a major conclusion of this paper, but it is not very surprising and I don't see how it's very useful. The perspective of language emergence literature is presented a bit strangely. The self-play to supervision baseline seems to be presented as an approach from the language emergence literature. I don't think this is what any of that literature promotes exactly, though it is close. Generally, I (and likely others) don't think it's too surprising that trying to fine-tune a self-play model with language supervision data doesn't work very well, for the same reasons cited in this paper (point 3 of section 5). I think the general strategy when trying to gain practical benefits from self-play pre-training is a translation approach where the learned language is translated into a known language like English rather than trying to directly align it to English as does the supervision approach in this paper. This particular baseline would be more useful if the paper considered learning some kind of translation layer on top of the self-play pre-trained model.
* How significant are the performance differences in figure 7a, especially those between the frozen and non-frozen models? Is the frozen model really better or this performance difference just due to noise?
* I'm somewhat skeptical that these trends will generalize to other tasks/models. The main goal of this paper is to inform future work. That makes it even more important than normal that the trends identified here are likely to generalize well. Are these trends likely to generalize well? Does the paper address when these trends are expected to hold anywhere?
Minor Presentation Weaknesses:
* Figure 4: I think the sub-figures are mis-labeled in the caption.
* In the related work I'm not sure the concept of generations is right. I think it should refer to different languages of different agents across time rather than different languages of the same agent across time.
Missing details / clarification questions:
* What exactly does Figure 4c compare? Are both methods distilled from ensembles or is the blue line normal S2P while the other is distilled from an ensemble of compositional languages? It's not clear since point (3) in section 5 refers to the S2P result (not Pop-S2P) in that plot. I'm also assuming that PB-S2P means the same thing as Pop-S2P, but that's not made clear anywhere. Does PB stand for Population Based?
* In the rand setting how is convergence defined? Do both objectives need to converge or just one?
* In the sched_rand_frz setting what is r?
* In the IBR how are the distractor images picked?
Suggestions:
* Can't both self-play and supervision be used at the same time (just use a weighted combination of the two objectives)? I don't think the paper ever did this but it seems like a very useful variation to consider.
Preliminary Evaluation
---
Clarity: The writing is fairly clear, though some details are lacking.
Significance: This work could help inspire some future models in the language emergence literature.
Quality: Experiments are aligned with the paper's goals and support its conclusions.
Originality: The distillation approach and curricula are novel.
Overall the work could prove to be an interesting and useful reference point inside the language emergence literature so I recommend it for acceptance.