  This paper presents an extension of the Mirror Descent method to overcome
challenges in cooperative Multi-Agent Reinforcement Learning (MARL) settings,
where agents have varying abilities and individual policies. The proposed
Heterogeneous-Agent Mirror Descent Policy Optimization (HAMDPO) algorithm
utilizes the multi-agent advantage decomposition lemma to enable efficient
policy updates for each agent while ensuring overall performance improvements.
By iteratively updating agent policies through an approximate solution of the
trust-region problem, HAMDPO guarantees stability and improves performance.
Moreover, the HAMDPO algorithm is capable of handling both continuous and
discrete action spaces for heterogeneous agents in various MARL problems. We
evaluate HAMDPO on Multi-Agent MuJoCo and StarCraftII tasks, demonstrating its
superiority over state-of-the-art algorithms such as HATRPO and HAPPO. These
results suggest that HAMDPO is a promising approach for solving cooperative
MARL problems and could potentially be extended to address other challenging
problems in the field of MARL.

Machine Learning;Artificial Intelligence;Multiagent Systems