Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge.
The paper presents SAD (Simplified Action Decoder), a new method to address an issue in Centralized Training / Decentralized Control regimes: that exploratory actions during training can make those actions less informative for the agents who observe and attempt to learn from it. The method addresses this issue by allowing the agent to select two actions: one (the possibly exploratory action) is applied to the environment and the other (the greedy action) is presented to other agents as part of their observations for the next turn. The authors use a distributed recurrent DQN architecture and apply the resulting agent to a toy problem and to the Hanabi Learning Environment. The authors claim that the method offers improvement over the Bayesian Action Decoder (BAD), that has been similarly applied to the same environments, being simpler, more sample efficient and achieving overall higher scores, which is confirmed by their results: the agent outperforms BAD in 2-player Hanabi (where BAD was previously state-of-the-art) and the best scores out of any learning agent (although not as high as some non-learning agents such as WTFWThat in the 3-5 player versions. The paper does not directly address the ad-hoc cooperation aspect of Hanabi, and it is unclear wheter the method could be used as-is for that problem, due to its reliance on centralized training. Nevertheless, the paper represents a relevant improvement to the self-play aspect of the game, and the core insight that the method leverages could conceivably be applied to minimize the noise introduced by exploration in other cooperative CT/DC problems. For this reason, I recommend the paper to be accepted. Typos/language issues: Introduction: “spend vast amounts of time coordinate” -> coordinating Section 3.1: “While our method are general” -> methods *accomplish a state of the art in Hanabi* I see what you mean but this is a strange phrasing.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning
In recent years we have seen fast progress on a number of benchmark problems in AI, with modern methods achieving near or super human performance in Go, Poker and Dota. One common aspect of all of these challenges is that they are by design adversarial or, technically speaking, zero-sum. In contrast to these settings, success in the real world commonly requires humans to collaborate and communicate with others, in settings that are, at least partially, cooperative. In the last year, the card game Hanabi has been established as a new benchmark environment for AI to fill this gap. In particular, Hanabi is interesting to humans since it is entirely focused on theory of mind, i.e. the ability to effectively reason over the intentions, beliefs and point of view of other agents when observing their actions. Learning to be informative when observed by others is an interesting challenge for Reinforcement Learning (RL): Fundamentally, RL requires agents to explore in order to discover good policies. However, when done naively, this randomness will inherently make their actions less informative to others during training. We present a new deep multi-agent RL method, the Simplified Action Decoder (SAD), which resolves this contradiction exploiting the centralized training phase. During training SAD allows other agents to not only observe the (exploratory) action chosen, but agents instead also observe the greedy action of their team mates. By combining this simple intuition with an auxiliary task for state prediction and best practices for multi-agent learning, SAD establishes a new state of the art for 2-5 players on the self-play part of the Hanabi challenge.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper presents SAD (Simplified Action Decoder), a new method to address an issue in Centralized Training / Decentralized Control regimes: that exploratory actions during training can make those actions less informative for the agents who observe and attempt to learn from it. The method addresses this issue by allowing the agent to select two actions: one (the possibly exploratory action) is applied to the environment and the other (the greedy action) is presented to other agents as part of their observations for the next turn. The authors use a distributed recurrent DQN architecture and apply the resulting agent to a toy problem and to the Hanabi Learning Environment. The authors claim that the method offers improvement over the Bayesian Action Decoder (BAD), that has been similarly applied to the same environments, being simpler, more sample efficient and achieving overall higher scores, which is confirmed by their results: the agent outperforms BAD in 2-player Hanabi (where BAD was previously state-of-the-art) and the best scores out of any learning agent (although not as high as some non-learning agents such as WTFWThat in the 3-5 player versions. The paper does not directly address the ad-hoc cooperation aspect of Hanabi, and it is unclear wheter the method could be used as-is for that problem, due to its reliance on centralized training. Nevertheless, the paper represents a relevant improvement to the self-play aspect of the game, and the core insight that the method leverages could conceivably be applied to minimize the noise introduced by exploration in other cooperative CT/DC problems. For this reason, I recommend the paper to be accepted. Typos/language issues: Introduction: “spend vast amounts of time coordinate” -> coordinating Section 3.1: “While our method are general” -> methods *accomplish a state of the art in Hanabi* I see what you mean but this is a strange phrasing.
assistant
The paper presents SAD (Simplified Action Decoder), a new method to address an issue in Centralized Training / Decentralized Control regimes: that exploratory actions during training can make those actions less informative for the agents who observe and attempt to learn from it. The method addresses this issue by allowing the agent to select two actions: one (the possibly exploratory action) is applied to the environment and the other (the greedy action) is presented to other agents as part of their observations for the next turn. The authors use a distributed recurrent DQN architecture and apply the resulting agent to a toy problem and to the Hanabi Learning Environment. The authors claim that the method offers improvement over the Bayesian Action Decoder (BAD), that has been similarly applied to the same environments, being simpler, more sample efficient and achieving overall higher scores, which is confirmed by their results: the agent outperforms BAD in 2-player Hanabi (where BAD was previously state-of-the-art) and the best scores out of any learning agent (although not as high as some non-learning agents such as WTFWThat in the 3-5 player versions. The paper does not directly address the ad-hoc cooperation aspect of Hanabi, and it is unclear wheter the method could be used as-is for that problem, due to its reliance on centralized training. Nevertheless, the paper represents a relevant improvement to the self-play aspect of the game, and the core insight that the method leverages could conceivably be applied to minimize the noise introduced by exploration in other cooperative CT/DC problems. For this reason, I recommend the paper to be accepted. Typos/language issues: Introduction: “spend vast amounts of time coordinate” -> coordinating Section 3.1: “While our method are general” -> methods *accomplish a state of the art in Hanabi* I see what you mean but this is a strange phrasing.