Actor-Critic Policy Optimization in Partially Observable Multiagent Environments
Optimization of parameterized policies for reinforcement learning (RL) is an important and challenging problem in artificial intelligence. Among the most common approaches are algorithms based on gradient ascent of a score function representing discounted return. In this paper, we examine the role of these policy gradient and actor-critic algorithms in partially-observable multiagent environments. We show several candidate policy update rules and relate them to a foundation of regret minimization and multiagent learning techniques for the one-shot and tabular cases, leading to previously unknown convergence guarantees. We apply our method to model-free multiagent reinforcement learning in adversarial sequential decision problems (zero-sum imperfect information games), using RL-style function approximation. We evaluate on commonly used benchmark Poker domains, showing performance against fixed policies and empirical convergence to approximate Nash equilibria in self-play with rates similar to or better than a baseline model-free algorithm for zero-sum games, without any domain-specific state space reductions.
2. The paper draws the connection between regret minimization and actor-critic algorithms. Specifically, it shows the connection by defining a new variant of an actor-critic algorithm that performs an exhaustive policy evaluation at each stage (denoted as policy-iteration-actor-critic), together with an adaptive learning rate. Then, under this setting, it is said that the actor-critic algorithm basically minimizes regret and converges to a Nash equilibrium. The paper suggests a few new versions of policy gradient update rules (Q-based Policy Gradient, Regret Policy Gradient, and Regret Matching Policy Gradient) and evaluates them on multi-agent zero-sum imperfect information games. To my understanding, Q-Based Policy Gradient is basically an advantage actor-critic algorithm (up to a transformation of the learned baseline) 3. The authors mention a “reasonable parameter sweep” over the hyperparameters. I’m curious to know the stability of the proposed actor-critic algorithms over the different trials 4. The paper should be proofread again. Many broken sentences and typos (“…be the reach probability of the all policies action choices…”, “…Two transition buffers of are used…”) 5. G_t is not defined. I believe the meaning is cumulative reward R_t, since the authors use them interchangeably.  6. Being at the border between game theory and RL makes notation very confusing. For example, pi that stands for policy (a.k.a strategy in game theory literature) denotes “transition probability” in game theory. Therefore, I encourage the authors to include a table of notations. 7. In line 148 the authors give a fresh intuition about the policy gradient expression (“…so estimate of the gradient must be corrected by dividing by policy probability”). However, that’s counterintuitive. A sampled trajectory with low probabilities would produce a high-intensity gradient (dividing by small numbers), which is the opposite of what is meant to happen. Please clarify your intuition. 8. In line 150 you define Q-based policy gradient, which basically has the form of advantage actor-critic. If this is correct, then please illuminate the relation for the sake of clarity (little or none knows what Q-based PG is vs the popularity of advantage actor-critic methods) 9. Please explain the untraditional containment symbol of ha and z in line 187. Did you mean to indicate set containment? And please clarify what is “ha” (you’ve defined “eta of ha and z” but never “ha” explicitly) 10. Do you share the same critic for all players?  Summary:  Quality - the paper draws a connection between Q values in RL and counterfactual values from the field of game theory. Second, it draws a connection between actor-critic and regret minimization. Although this is true under a very specific setup, I believe that this is an important result that connects two worlds that have been parallel up to now.  Clarity - The paper is hard to follow because it stands on the borderline between disciplines with conflicting notations. In order to remedy the problem, the paper should pay extra attention to notations and their use. Besides that, an extra proofreading of the paper would benefit the paper significantly (many broken sentences and typos)  Originality and significance - I believe that the paper is novel and significant to a fair degree