In cooperative multi-agent reinforcement learning (MARL), how to design a suitable reward signal to accelerate learning and stabilize convergence is a critical problem. The global reward signal assigns the same global reward to all agents without distinguishing their contributions, while the local reward signal provides different local rewards to each agent based solely on individual behavior. Both of the two reward assignment approaches have some shortcomings: the former might encourage lazy agents, while the latter might produce selfish agents. In this paper, we study reward design problem in cooperative MARL based on packet routing environments. Firstly, we show that the above two reward signals are prone to produce suboptimal policies. Then, inspired by some observations and considerations, we design some mixed reward signals, which are off-the-shelf to learn better policies. Finally, we turn the mixed reward signals into the adaptive counterparts, which achieve best results in our experiments. Other reward signals are also discussed in this paper. As reward design is a very fundamental problem in RL and especially in MARL, we hope that MARL researchers can rethink the rewards used in their systems.
The authors suggest using a mixture of shared and individual rewards within a MARL environment to induce cooperation among independent agents. They show that on their specific application this can lead to a better overall global performance than purely sharing the global signal, or using just the independent rewards.
The paper is a little too focused on the packet routing example domain and fails to deliver much in terms of a general theory of reward design for cooperative behaviours beyond showing that mixed rewards can lead to improved results in their domain. They discuss what and how rewards, and this could be made more formal, as well as (at the very least) some guiding principles to follow when mixing rewards. It feels like there is a missing section between sections 2 and 3, where this methodological content could be described.
The rest of the paper has similar issues, with key intuition and concepts either missing entirely or under-represented. The technical content often assumes that the reader is familiar with certain terms, and it is difficult to see what meaningful conclusions can be drawn from the evaluation.
On a minor note, the use of the term cooperative in this paper could be better defined. In game theory, cooperative games are those in which agents share rewards. Non-cooperative (game theory) games are those where agents have general reward signals (not necessarily cooperature or adversarial). Conventionally (yes there is existing reward design/shaping literature for MARL) people have used the same terms in MARL. Perhaps the authors could define their approach as weakly cooperative, or emergent cooperation.
The related work could be better described. There are existing papers on MARL and the issues with cooperation among independent learners, and this could be referenced. This includes reward shaping and reward potential. I would also have expected to see brief mention of empowerment in this section too (the agent favouring states where it has the power to control outcomes in an information theoretic sense), as an underyling principle for intrinsic reward. However, more importantly, the authors really needed to do more to synthesize this into an overall picture of what principles are at play and what ideas/methods exist that have tried to exploit some of these principles.
Detailed comments:
  • [p2] the authors say "We set the meta reward signals as 1 - max(U l ).", before they define what U_l is.
  • [p2] we have "As many applications in the real world can be modeled using similar
methods, we expect that other fields can also benefit from this work." This statement is too vague, and the authors could do more to identify which application areas might benefit.
  • [p3, first para] "However, the reward design studies for MARL is so limited." Drop the word 'so'. Also, I would argue that there have been quite a few (non-deep) discussions about reward design in MARL, cooperative, non-cooperative and competitive domains. 
  • [p3, sec 2.2] "This makes the diligent agents confuse about..." should be "confused", and I would advise against anthropomorphism at least when the meaning is obscured.
  • [p3, sec 3] "After having considered several other options, we finally choose the Packet Routing Domain as our experimental environments." Not sure what useful information is being conveyed here.
  • [sec 3] THe domain could be better described with intuition and formal descriptions, e.g. link utilization ratio, etc, before.
  • [p6] "Importantly, the proposed blR seems to have similar capacity with dlR," The discussion here is all in terms of the reward acronyms with very little call on intuition or other such assistance to the reader.
  • [p7] "We firstly try gR without any thinking" The language could be better here.