  Although multi-agent reinforcement learning can tackle systems of
strategically interacting entities, it currently fails in scalability and lacks
rigorous convergence guarantees. Crucially, learning in multi-agent systems can
become intractable due to the explosion in the size of the state-action space
as the number of agents increases. In this paper, we propose a method for
computing closed-loop optimal policies in multi-agent systems that scales
independently of the number of agents. This allows us to show, for the first
time, successful convergence to optimal behaviour in systems with an unbounded
number of interacting adaptive learners. Studying the asymptotic regime of
N-player stochastic games, we devise a learning protocol that is guaranteed to
converge to equilibrium policies even when the number of agents is extremely
large. Our method is model-free and completely decentralised so that each agent
need only observe its local state information and its realised rewards. We
validate these theoretical results by showing convergence to Nash-equilibrium
policies in applications from economics and control theory with thousands of
strategically interacting agents.

Multiagent Systems