When communicating, humans rely on internally-consistent language representations. That is, as speakers, we expect listeners to behave the same way we do when we listen. This work proposes several methods for encouraging such internalconsistency in dialog agents in an emergent communication setting: shared embeddings, symmetric encoding/decoding, and self-play. We consider two hypotheses about the effect of internal-consistency constraints: 1) that they improve agents' ability to refer to unseen referents, and 2) that they improve agents' ability to generalize across communicative roles (e.g. performing as a speaker despite only being trained as a listener). While we do not find evidence in favor of the former, our results show significant support for the latter. In particular, we show that, when self-consistency is enforced via self-play, agents are able to perform in novel roles as well as if they were trained with direct supervision in those roles.
The paper analyzes if enforcing internal-consistency for speaker-listener setup can (i) improve the ability of the agents to refer to unseen referents (ii) generalize for different communicative roles. The paper evaluates a transformer and arecurrent model modified with various sharing strategies on a single-turn reference game. Finally, the paper claims that results with self-play suggest that internal consistency doesn’t help (i) but improves cross-role performance for (ii).
As a reader, the paper doesn’t provide me a concrete finding which can help in designing future multi-agent systems. Most of the results for the experiments (except self-play) don’t have a uniform signal across the board to deduce whether the internal-consistency works in all of the cases. Most of the speaker-listener scenarios emerge in dialog based settings which are multi-turn and require agents to act both as speaker and listener. Though paper advocates through some of its results that self-play is helpful in generalization across roles via internal-consistency, without multi-step experiments, qualitative and quantitative analysis of what is happening and why there is so much variation, the paper is weak in its current form. Therefore, I recommend weak reject as my rating. Below I discuss some of the concerns in detail:
Without multi-step evaluation, it is hard to gauge the extent to which self-play for internal consistency help in generalization of the roles. For e.g., task from Das et. al. (2017) [1] provides a clear signal on how well the agents are able to communicate through dialog evaluation. So in 5.2.1, the setup which requires training in both roles can provide better signal overall if it was trained to do multi-step conversation.
Paper is missing any kind of quantitative or qualitative analysis. What are the differences between the embeddings of the agent that learned via self-play and the one which learned directly. It also be interesting to see how the shared embeddings and symmetric encoding and decoding affect these embedding and might help explain the drop and randomness. In Table 4., the results on symmetric encoding suggest that the claim of generalization through internal consistency might not hold everywhere. For Shared Embedding results, on RNN shapes, it is surprising that training in one role improves performance through internal consistency while in both roles it drops. These require further analysis to solidify the claim. Given the flaky results, to boost the claim, have authors tried other settings to test internal-consistency like Predator-Prey?
Things that didn’t affect the score:
Related work section is missing the relevant discussion on continuous communication work and discussion on why internal consistency wasn’t tested on those settings as well. (See Singh et.al [2]., Sukhbaatar et.al. [3], Das et.al. [4] etc)
The number of pages are above eight, you should reduce the redundancy between table descriptions and text and maybe squeeze Section 2, decrease setup explanation.
The setup for training and test sets explained at the end of page 7 isn’t very clear to me and needs to be rephrased.
[1] Das, Abhishek, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. "Learning cooperative visual dialog agents with deep reinforcement learning." In Proceedings of the IEEE International Conference on Computer Vision, pp. 2951-2960. 2017.
[2] Sukhbaatar, Sainbayar, and Rob Fergus. "Learning multiagent communication with backpropagation." In Advances in Neural Information Processing Systems, pp. 2244-2252. 2016.
[3] Singh, Amanpreet, Tushar Jain, and Sainbayar Sukhbaatar. "Learning when to communicate at scale in multiagent cooperative and competitive tasks." arXiv preprint arXiv:1812.09755 (2018).
[4] Das, Abhishek, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, and Joelle Pineau. "Tarmac: Targeted multi-agent communication." arXiv preprint arXiv:1810.11187 (2018).
=========
Post-rebuttal Comments
=========
Thanks for updating the manuscript to resolve my and R2's concerns. The new analysis section does provide good insights into what exactly is happening.
When I was talking about actionable insights, I was talking about both negative and positive insights. Currently, the only take-away is that self-play helps in generalizing to listener roles as well. For the other negative insight that internal consistency doesn't help with generalization, as R2 suggested, it is unclear why that would be case in the first place (I read the pscyhology arguments, but I am not still not convinced). I still believe that without multi-step communication, the work is as useful as it can be in current form. In real world, no meaningful conversation is usually one step.
For Predator-Prey setup, I was talking about OpenAI https://github.com/openai/multiagent-particle-envs in which multiple tasks can be setup. For e.g. if prey thinks of what action predator might take, does internal consistency help prey to perform better?
I think most of what you got is correct for multi-step, see second para for more details in this response.
Thanks for bringing the manuscript under 8 pages. [3] is still missing from references.
Final comments: I would like to see multi-step experiments due to the reasons I explained above. The scheme of internal-consistency should be applicable beyond conversation to Predator-Prey setups also, thus, I feel experiments are not enough (only on 1 setting) to claim generalization of the hypothesis. Beyond these comments, I feel this is still a step in right direction and I would like to update my rating to weak accept while hoping that authors try to address these issues in camera-ready version if paper gets accepted.