Mean field games (MFG) facilitate the otherwise intractable reinforcement learning (RL) in large-scale multi-agent systems (MAS) , through reducing interplays among agents to those between a representative individual agent and the mass of the population. While , RL agents are notoriously prone to unexpected behaviours due to  reward mis-specification. This problem is exacerbated by an expanding scale of MAS. Inverse reinforcement learning (IRL) provides a framework to automatically acquire proper reward functions from expert demonstrations. Extending IRL to MFG , however, is challenging due to the complex notion of mean-field-type equilibria and the coupling between agent-level and population-level dynamics. To this end, we propose mean field inverse reinforcement learning (MFIRL), a novel model-free IRL framework for MFG . We derive the algorithm based on a new equilibrium concept that incorporates entropy regularization, and the maximum entropy IRL framework. Experimental results on simulated environments demonstrate that MFIRL is sample efficient and can accurately recover the ground-truth reward functions , compared to the state-of-the-art method.
Mean field games (MFG) facilitate the application of reinforcement learning (RL) in large-scale multi-agent systems , through reducing interplays among agents to those between an individual agent and the average effect from the population. However , RL agents are notoriously prone to unexpected behaviours due to the reward mis-specification. Although inverse RL (IRL) holds promise for automatically acquiring suitable rewards from demonstrations, its extension to MFG is challenging due to the complicated notion of mean-field-type equilibria and the coupling between agent-level and population-level dynamics. To this end, we propose a novel IRL framework for MFG , called Mean Field IRL (MFIRL), where we build upon a new equilibrium concept and the maximum entropy IRL framework. Crucially, MFIRL is brought forward as the first IRL method that can recover the agent-level ( ground-truth ) reward functions for MFG. Experiments show the superior performance of MFIRL on sample efficiency, reward recovery and robustness against varying environment dynamics , compared to the state-of-the-art method.
R
otherwise intractable
application of
D
(MAS)
R
a representative
an
R
mass of
average effect from
R
While
However
A
the
R
This problem is exacerbated by an expanding scale of MAS. Inverse reinforcement learning
Although inverse RL
R
provides a framework to automatically acquire proper reward functions from expert demonstrations. Extending IRL to MFG , however,
holds promise for automatically acquiring suitable rewards from demonstrations, its extension to MFG
R
complex
complicated
R
mean field inverse reinforcement learning (MFIRL), a novel model-free
a novel
R
. We derive the algorithm based on
, called Mean Field IRL (MFIRL), where we build upon
D
that incorporates entropy regularization,
R
Experimental results on simulated environments demonstrate that MFIRL is sample efficient and can accurately recover the
Crucially, MFIRL is brought forward as the first IRL method that can recover the agent-level (
R
reward functions
) reward functions for MFG. Experiments show the superior performance of MFIRL on sample efficiency, reward recovery and robustness against varying environment dynamics