Multiagent reinforcement learning (MARL) attempts to optimize policies of intelligent agents interacting in the same environment. However, it may fail to converge to a Nash equilibrium in some games. We study independent MARL under the more demanding solution concept of iterated elimination of strictly dominated strategies. In dominance solvable games, if players iteratively eliminate strictly dominated strategies until no further strategies can be eliminated, we obtain a single strategy profile. We show that convergence to the iterated dominance solution is guaranteed for several reinforcement learning algorithms (for multiple independent learners). We illustrate an application of our results by studying mechanism design for principal-agent problems, where a principal wishes to incentivize agents to exert costly effort in a joint project when it can only observe whether the project succeeded, but not whether agents actually exerted effort. We show that MARL converges to the desired outcome if the rewards are designed so that exerting effort is the iterated dominance solution, but fails if it is merely a Nash equilibrium.
This paper studies reinforcement learning algorithms in a specific subset of multi-agent environments that are 'dominance solvable'. This means that, given an initial set of strategies in the game, if we iteratively remove 'dominated strategies' (those whose utility is strictly less than another strategy independent of the strategies used by other agents), then only one strategy remains for each player. The remaining strategy is called the iterated dominance solution. The paper proves the convergence of certain RL algorithms (REINFORCE in the 2-action case, and importance weighted monte-carlo policy iteration in the multi-action case) for normal-form games. The paper demonstrates the utility of this via mechanism design: in a principal-agent problem where one can design the rewarding scheme given by a 'principal agent' to various (RL) sub-agents, rewarding schemes motivated by iterated dominance guarantees the best solution for the principal agent, whereas schemes motivated by Nash equilibria do not. 
The paper is quite well-written and understandable. To my knowledge, the idea is novel and has not yet been explored in the RL literature (UPDATE: based on Reviewer #1's review, this may not be the case. I'll wait to hear the author response to this). I did not check the proofs thoroughly. However, the experiments in the principal-agent problem make sense, and it's interesting to see that iterated dominance reward schemes results in good performance for the principal agent. I appreciate that, while the main results in the paper are limited to normal-form games (which are quite restricted), there are empirical results in the appendix showing the extension to Markov games with multiple timesteps, suggesting that the applicability of iterated dominance reward schemes extend beyond the simple two-action case, where no temporally extended decisions need to be made. Even so, the Markov game considered is fairly simplistic. 
My personal curiosity about this paper revolves around scaling to real-world applications. This is not really discussed in the paper; the conclusion talks about directions for future work, for example expanding the number of RL algorithms where convergence can be proven, or producing complexity bounds for convergence. What I want to know is: what sorts of games can we compute the iterated dominance reward schemes for? How can this be applied when the space of policies becomes too large to be enumerated (and thus determining whether a policy is strictly dominated becomes impossible)? I don't expect this paper to solve these issues, but it would be nice to have a discussion of them. 
Overall, I'd say this paper is interesting to the multi-agent RL community and I could imagine others building off of this work, so I err on the side of acceptance. 
Small fixes:
- Our proof of Theorem 3.1 -> Theorem 3.2
- I'd recommend extending the captions of figures 6-8 and 9-11 in the Appendix. 
- Close bracket in Section 6.3 title