Intrinsic Social Motivation via Causal Influence in Multi-Agent RL
We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent*s actions, where causal influence is assessed using counterfactual reasoning. The reward does not depend on observing another agent*s reward function, and is thus a more realistic approach to MARL than taken in previous work. We show that the causal influence reward is related to maximizing the mutual information between agents* actions. We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward. Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols. Therefore, we also employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward. Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents. This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents.
 This paper proposes an approach to model social influence in a scenario-independent manner by instantiating the concept of intrinsic motivation and combine various human abilities as part of a reinforcement learning function in order to improve the agent*s operation in social dilemma scenarios. Agents are operationalised as convolutional neural network, linear layers and LSTM. Using these base mechanisms, different abilities (communication, models of other agents (MOA)), their causal influence is inferred based on counterfactual actions. The architecture is explored across two different sequential social dilemmas. The architecture is described in sufficient detail, with particular focus on the isolation of causal influence for communication and MOA influence. The experimental evaluation is described in sufficient detail, given the low complexity of the scenarios. While the agents with communicative ability and MOA show superior performance, a few results warrant clarification. Figure 6a) highlights the performance of influencers in contrast to a visible actions baseline. This specific scenarios shows the necessity to run experiments for larger number of runs, since it appears that action observations may actually outperform influencer performance beyond 3 steps. Please clarify what is happening in this specific case, and secondly, justify your choice of steps used in the experimental evaluation. Another results that requires clarification is Figure 6f), which is not sufficiently discussed in the text, yet provides interesting patterns between the MOA baseline performance decaying abruptly at around 3 steps, with the influence MOA variant only peaking after that. Please clarify the observation. Also, could you draw conclusions or directions for a combination of the different approaches to maximise the performance (more generally, beyond this specific observation)? A valuable discussion is the exemplification of specific agent behaviour on Page 7. While it clarifies the signalling of resources in this specific case, it also shows shortcomings of the model*s realism. How would the model perform if agents had limited resources and would die upon depletion (e.g. the de facto altruistic influencer in this scenario - since it only performs two distinct actions)? The extent of generalisability should be considered in the discussion. In general, the paper motivates and discusses the underlying work in great detail and is written in an accessible manner (minor comment: the acronym LSTM is not explicitly introduced). The quality of presentation is good.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Intrinsic Social Motivation via Causal Influence in Multi-Agent RL
We derive a new intrinsic social motivation for multi-agent reinforcement learning (MARL), in which agents are rewarded for having causal influence over another agent*s actions, where causal influence is assessed using counterfactual reasoning. The reward does not depend on observing another agent*s reward function, and is thus a more realistic approach to MARL than taken in previous work. We show that the causal influence reward is related to maximizing the mutual information between agents* actions. We test the approach in challenging social dilemma environments, where it consistently leads to enhanced cooperation between agents and higher collective reward. Moreover, we find that rewarding influence can lead agents to develop emergent communication protocols. Therefore, we also employ influence to train agents to use an explicit communication channel, and find that it leads to more effective communication and higher collective reward. Finally, we show that influence can be computed by equipping each agent with an internal model that predicts the actions of other agents. This allows the social influence reward to be computed without the use of a centralised controller, and as such represents a significantly more general and scalable inductive bias for MARL with independent agents.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes an approach to model social influence in a scenario-independent manner by instantiating the concept of intrinsic motivation and combine various human abilities as part of a reinforcement learning function in order to improve the agent*s operation in social dilemma scenarios. Agents are operationalised as convolutional neural network, linear layers and LSTM. Using these base mechanisms, different abilities (communication, models of other agents (MOA)), their causal influence is inferred based on counterfactual actions. The architecture is explored across two different sequential social dilemmas. The architecture is described in sufficient detail, with particular focus on the isolation of causal influence for communication and MOA influence. The experimental evaluation is described in sufficient detail, given the low complexity of the scenarios. While the agents with communicative ability and MOA show superior performance, a few results warrant clarification. Figure 6a) highlights the performance of influencers in contrast to a visible actions baseline. This specific scenarios shows the necessity to run experiments for larger number of runs, since it appears that action observations may actually outperform influencer performance beyond 3 steps. Please clarify what is happening in this specific case, and secondly, justify your choice of steps used in the experimental evaluation. Another results that requires clarification is Figure 6f), which is not sufficiently discussed in the text, yet provides interesting patterns between the MOA baseline performance decaying abruptly at around 3 steps, with the influence MOA variant only peaking after that. Please clarify the observation. Also, could you draw conclusions or directions for a combination of the different approaches to maximise the performance (more generally, beyond this specific observation)? A valuable discussion is the exemplification of specific agent behaviour on Page 7. While it clarifies the signalling of resources in this specific case, it also shows shortcomings of the model*s realism. How would the model perform if agents had limited resources and would die upon depletion (e.g. the de facto altruistic influencer in this scenario - since it only performs two distinct actions)? The extent of generalisability should be considered in the discussion. In general, the paper motivates and discusses the underlying work in great detail and is written in an accessible manner (minor comment: the acronym LSTM is not explicitly introduced). The quality of presentation is good.
assistant
 This paper proposes an approach to model social influence in a scenario-independent manner by instantiating the concept of intrinsic motivation and combine various human abilities as part of a reinforcement learning function in order to improve the agent*s operation in social dilemma scenarios. Agents are operationalised as convolutional neural network, linear layers and LSTM. Using these base mechanisms, different abilities (communication, models of other agents (MOA)), their causal influence is inferred based on counterfactual actions. The architecture is explored across two different sequential social dilemmas. The architecture is described in sufficient detail, with particular focus on the isolation of causal influence for communication and MOA influence. The experimental evaluation is described in sufficient detail, given the low complexity of the scenarios. While the agents with communicative ability and MOA show superior performance, a few results warrant clarification. Figure 6a) highlights the performance of influencers in contrast to a visible actions baseline. This specific scenarios shows the necessity to run experiments for larger number of runs, since it appears that action observations may actually outperform influencer performance beyond 3 steps. Please clarify what is happening in this specific case, and secondly, justify your choice of steps used in the experimental evaluation. Another results that requires clarification is Figure 6f), which is not sufficiently discussed in the text, yet provides interesting patterns between the MOA baseline performance decaying abruptly at around 3 steps, with the influence MOA variant only peaking after that. Please clarify the observation. Also, could you draw conclusions or directions for a combination of the different approaches to maximise the performance (more generally, beyond this specific observation)? A valuable discussion is the exemplification of specific agent behaviour on Page 7. While it clarifies the signalling of resources in this specific case, it also shows shortcomings of the model*s realism. How would the model perform if agents had limited resources and would die upon depletion (e.g. the de facto altruistic influencer in this scenario - since it only performs two distinct actions)? The extent of generalisability should be considered in the discussion. In general, the paper motivates and discusses the underlying work in great detail and is written in an accessible manner (minor comment: the acronym LSTM is not explicitly introduced). The quality of presentation is good.