RELATIONAL FORWARD MODELS FOR MULTI-AGENT LEARNING
The behavioral dynamics of multi-agent systems have a rich and orderly structure, which can be leveraged to understand these systems, and to improve how artificial agents learn to operate in them. Here we introduce Relational Forward Models (RFM) for multi-agent learning, networks that can learn to make accurate predictions of agents' future behavior in multi-agent environments. Because these models operate on the discrete entities and relations present in the environment, they produce interpretable intermediate representations which offer insights into what drives agents' behavior, and what events mediate the intensity and valence of social interactions. Furthermore, we show that embedding RFM modules inside agents results in faster learning systems compared to non-augmented baselines. As more and more of the autonomous systems we develop and interact with become multi-agent in nature, developing richer analysis tools for characterizing how and why agents make decisions is increasingly necessary. Moreover, developing artificial agents that quickly and safely learn to coordinate with one another, and with humans in shared environments, is crucial.
This paper studies predicting multi-agent behavior using a proposed neural network architecture. The architecture, called a relational forward model (RFM) is the same graph network proposed by Battaglia et al., 2018, but adds a recurrent component. Two tasks are define: predict the next action of each agent, and predict the sum of future rewards. The paper demonstrates that RFMs outperform two baselines and two ablations. The authors also show that edge activation magnitudes are correlated with certain phenomenons (e.g. an agent walking towards an entity, or an entity being “on” or “off”). The authors also show that appending the output of a pre-trained RFM to the state of a policy can help it learn faster.
Overall, this paper presents some interesting ideas and is easy to follow, but the significance of the paper is not clear. The architecture is a rather straightforward extensions of previous work, and using graph networks for predictive modeling in multi-agent settings has been examined in the past, making the technical contributions not particularly novel. Examining the correlation between edge activation magnitudes and certain events is intriguing and perhaps the most novel aspect of this paper, but it is not clear how or why this information would be useful. There a few unsubstantiated claims that are concerning. There are also some odd experimental decisions and results that should be addressed.
For specific comments:
1. Why would using a recurrent network help (i.e. RFM vs Feedforward)? Unless the policies are non-Markovian, the entire prediction problem should Markovian. I suspect that most of the gains are coming from the fact that the RFM method simply has more parameters than the Feedforward method (e.g. it can amortize some of the computation into the recurrent part of the network). Suggestion: train a Feedforward model that has more parameters (with appropriate hyperparameter sweeps) to see if this is the cause. If not, provide some analysis for why “memories of the relations between entities” would be any more beneficial than simply recomputing those relations.
2. The other potential reason that the recurrent method did better is that policy actions are highly correlated (e.g. because agents move in straight lines to locations). If so, then recurrent methods can outperform feedforward methods without having to learn anything about what actually causes policies to move in certain directions. Suggestion: measure the correlation between consecutive actions. If there is non-trivial correlation, than this suggests that RFM does better than Feedforward (which is basically prior work of Battaglia et. al.) is for the wrong reasons.
3. If I understand the evaluation metric correctly, for each rollout, it counts how many steps from the beginning of the rollout match perfectly before the first error occurs. Then it averages this “minimum time to failure” across all evaluation rollouts. If this is correct, why was this evaluation metric chosen? A much more natural metric would be to just compute the average number of errors on a test data-set (and if this is what is actually reported, please update the description to disambiguate the two). The current metric could be very deceptive:  Methods that do very well on states around the initial-state distribution but poorly near the end of trajectories (e.g. perfectly predicts the actions in the first 10 steps, but then resorts to random guessing for the last 99999 time steps) will outperform methods that have lower average error rate (e.g. a model that is correct 50% of the time). Suggestion: change the metrics to average number of errors, or report both, or provide a convincing argument why this metric is meaningful.
4. Unless I misunderstood, the results in Section 2.2.3 seem spurious and the claims seem unsubstantiated. For one, if we look at Equations (1) and (2), when we average over s_a1 and s_a2, they should both give the same average for R_a1. Put another way: the prune graph should (in theory) marginalize out s_a2. On average, its expected output should be the same as the output of the full graph (after marginalizing out s_a1 and s_a2). Obviously, it is possible to find specific rollouts where the full graph has higher value than the prune graph (and it seems Figure 4 does this), but it should equally be possible to find rollouts where the opposite is true. I’m hoping I misunderstood this section, but otherwise this seems to invalidate all the claims made in this section.
5. Even if concern #4 is addressed, the following sentence would still seem false: “This figure shows that teammates’ influence on each other during this time is beneficial to their return.” The figure simply shows predictions of the RFM, and not of the ground truth. Moreover, it’s not clear what “teammates’ influence” actually means.
6. The comparison to NRI seems rather odd, since that method uses strictly less information than RFM.
7. For Section 3, is the RFM module pretrained and then fine-tuned with the new policy? If so, this gives the “RFM + A2C” agent extra information indirectly via the pretrained weights of the RFM module.
8. I’m not sure what to make of the correlation analysis. It is not too surprising that there is some correlation (in fact, it’d be quite an interesting paper if the findings were that there wasn’t a correlation!), and it’s not clear to me how this could be used for debugging, visualizations, etc. If someone wanted to analyze the correlation between two entities and a policy’s action, it seems like they could directly model this correlation.
Some minor comments:
 - In Figure 3C, right, why isn’t the magnitude 0 at time=1? Based on the other plots in Figure 3c, it seems like it should be 0.
 - The month/year in many of the citations seems odd.
 - The use of the word “valence” seems unnecessarily flowery and distracting.
My main concern with this paper is that it is not particularly novel and the contribution seems questionable. I have some concerns over the experimental metric and Section 2.2.3, but even if that is clarified, it is not clear how impactful this paper would be. The use of a recurrent network seems unnecessary, unjustified, and not analyzed. The analysis of correlations is interesting, but not particularly compelling or surprising. And lastly, the RFM-augmented results are not very strong.
--
Edit: After discussing with the authors, I have changed my rating. The authors have adjusted some of the language, which I previously thought overstated the contributions and was misleading. They have added a number of experiments which valid the claim that their method is proposing a reasonable way of measuring collaboration. I also realized that I misunderstood one of the sections, and I encourage the authors to improve the presentation to (1) present the significance of the experiments more clear, (2) not overstate the results, and (3) emphasize the contribution more clearly.
Overall, the paper presents convincing evidence that factors in a graph neural networks do capture some notion of collaboration. I do not feel that the paper is particularly novel, but the experiments are thorough. Furthermore, their experiments show that adding an RFM module to an agent consistently helps (albeit not by much). Given that the multi-agent community is still trying to decide how to best quantify and use metrics for collaboration, I find it difficult to access the long-term impact of this paper. However, given the thoroughness of the experiments and analysis, I suspect that this will be valuable for the community and deserves some visibility.