Inequity aversion improves cooperation in intertemporal social dilemmas
Groups of humans are often able to find ways to cooperate with one another in complex, temporally extended social dilemmas. Models based on behavioral economics are only able to explain this phenomenon for unrealistic stateless matrix games. Recently, multi-agent reinforcement learning has been applied to generalize social dilemma problems to temporally and spatially extended Markov games. However, this has not yet generated an agent that learns to cooperate in social dilemmas as humans do. A key insight is that many, but not all, human individuals have inequity averse social preferences. This promotes a particular resolution of the matrix game social dilemma wherein inequity-averse individuals are personally pro-social and punish defectors. Here we extend this idea to Markov games and show that it promotes cooperation in several types of sequential social dilemma, via a profitable interaction with policy learnability. In particular, we find that inequity aversion improves temporal credit assignment for the important class of intertemporal social dilemmas. These results help explain how large-scale cooperation may emerge and persist.
Summary The authors consider the addition of inequity aversion to learning agents in social dilemmas. They show that inequity aversion as a heuristic helps systems converge to cooperation. They show this is true even when only a few agents are inequity averse.  Evaluation There is a lot to like about this paper. There are 2 main contributions: 1) Rather than considering end-to-end learning the authors consider simple heuristics that can be readily applied to solve a wide class of problems.  2) I also like the eligibility trace formulation of inequity aversion for Markov games, this opens, I think a wide way of experimenting with utility functions that include terms beyond one’s own reward in RL.  I recommend acceptance of this paper.   I have a few comments about the exposition for the authors that I think could improve the overall clarity of the paper.  First, I think it would be good to state in the introduction what the point of the research program is. Is it:  a) To design agents for environments where we control all agents, e.g. robot soccer (in which case, why not just optimize joint rewards?)  b) To design agents for environments where we control some but not all agents? (In this case, isn’t advantageous inequity aversion a bad idea since it makes one very exploitable?) c) To study how certain heuristics can evolve in a population? (In this case, it would be good to see the payoffs of each type of agent in games with mixed populations since that’s what would be favored by evolution) d) Something else?  I think nailing down more clearly in the introduction what the goal is and why people should care about this will make the paper even higher impact.