Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
Despite the success of single-agent reinforcement learning, multi-agent reinforcement learning (MARL) remains challenging due to complex interactions between agents. Motivated by decentralized applications such as sensor networks, swarm robotics, and power grids, we study policy evaluation in MARL, where agents with jointly observed state-action pairs and private local rewards collaborate to learn the value of a given policy. In this paper, we propose a double averaging scheme, where each agent iteratively performs averaging over both space and time to incorporate neighboring gradient information and local reward information, respectively. We prove that the proposed algorithm converges to the optimal solution at a global geometric rate. In particular, such an algorithm is built upon a primal-dual reformulation of the mean squared projected Bellman error minimization problem, which gives rise to a decentralized convex-concave saddle-point problem. To the best of our knowledge, the proposed double averaging primal-dual optimization algorithm is the first to achieve fast finite-time convergence on decentralized convex-concave saddle-point problems.
The article extends previous work of primal-dual optimisation for policy evaluation in RL to the distributed policy evaluation setting, maintaining attractive convergence rates for the extended algorithm. Overall, the article gradually builds its contribution and is reasonably easy to follow. A few exception to this are the start of related work, dropping citations in lists, and the lack of an explanation of the repeatedly mentioned 'convex-concave saddle-point problem'.       The authors equate averaging over 'agents' with averaging over 'space', which is somewhat of an imprecise metaphorical stretch in my view.       The contribution is honestly delineated (collaborative distributed policy evaluation with local rewards), and relevant related work is cited clearly. However, the innovation beyond related work is limited (averaging over agents is added). Empirical results show an improvement over two baselines, but a lag behind SAGA. From my understanding that algorithm has full central information, and is therefore given to indicate a sort of gap between the decentralised solution vs a centralised solver. This is not clear from the text.       Finally, the conclusion is called 'discussions' and has several language mistakes, which together with the shallow comments on the empirical performance puts an unsatisfactory end to this otherwise pleasant to read paper.              Minor remarks:     - 'settings with batch trajectory' -> -ies     - Assumption names like 'A1' are easy to miss, and might better be written out in full. Also, next to the first mention of A1 the citation is empty.     - J(w, \theta) and J(\theta, w) are inconsistently used around Equation 12.     - I could not spot what H1 refers to, when it is used after Equation 13. Maybe A1?