summaries: This paper presents AVEC, a new critic loss for model-free actor-critic Reinforcement Learning algorithms. The AVEC loss can be used with any actor-critic algorithm, with PPO, TRPO and SAC being evaluated in the paper. The loss builds on the mean-squared-error, and adds a term that minimizes $E_s [f_{\\phi}(s) - \\hat{V}^{\\pi_{\\theta_k}}(s) ]$. The addition of that extra term is motivated by recent research on the stability of actor-critic algorithms, and the benefits obtained by the AVEC loss are empirically demonstrated in numerous environments, with AVEC+PPO, AVEC+SAC and AVEC+TRPO. The paper explores an alternative loss function for fitting critic in Reinforcement Learning. Instead of using the standard mean squared loss between critic predictions and value estimates, the authors propose to use a loss function that also incorporates a variance term. The authors dub the approach AVEC. The authors combine their approach with popular RL algorithms such as SAC and PPO and evaluated on the standard benchmarks for continuous control. The paper proposes a simple and elegant idea for changing the value function objectives in deep RL and demonstrates reasonable empirical evidence of it's potential usefulness.  The authors also provide a clearly articulated intuitive motivation and provide experiments to support the proposal.  The idea complements several other algorithms and is therefore quite widely applicable (and easy to try). The analysis of the experiments is also quite interesting and clearly presented. 
abstractText: Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic uses a new state-value (resp. state-actionvalue) function approximation that learns the value of the states (resp. state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights.
authors.name: DIENTS USING
authors.name: RESIDUAL VARIANCE
authors.name: Yannis Flet-Berliac
authors.name: Reda Ouhamma
authors.name: Philippe Preux
references.authors: Marc Bellemare Sriram Srinivasan Georg Ostrovski Tom Schaul David Saxton Remi Munos
references.title: Unifying count-based exploration and intrinsic motivation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Erwin Coumans Yunfei Bai
references.title: Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016
references.authors: Yannis Flet-Berliac Philippe Preux
references.title: Merl: Multi-head reinforcement learning
references.venue: In Deep Reinforcement Learning Workshop,
references.authors: Yannis Flet-Berliac Philippe Preux
references.title: Only relevant information matters: Filtering out noisy samples to boost rl
references.venue: In International Joint Conference on Artificial Intelligence,
references.authors: Yannis Flet-Berliac Johan Ferret Olivier Pietquin Philippe Preux Matthieu Geist
references.title: Adversarially guided actor-critic
references.venue: In International Conference on Learning Representations,
references.authors: Evan Greensmith Peter L Bartlett Jonathan Baxter
references.title: Variance reduction techniques for gradient estimates in reinforcement learning
references.venue: Journal of Machine Learning Research,
references.authors: Shixiang Gu Timothy Lillicrap Ilya Sutskever Sergey Levine
references.title: Continuous deep q-learning with model-based acceleration
references.venue: In International Conference on Machine Learning,
references.authors: Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine
references.title: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
references.venue: In International Conference on Machine Learning,
references.authors: Andrew Ilyas Logan Engstrom Shibani Santurkar Dimitris Tsipras Firdaus Janoos Larry Rudolph Aleksander Madry
references.title: A closer look at deep policy gradients
references.venue: In International Conference on Learning Representations,
references.authors: Max Jaderberg Volodymyr Mnih Wojciech Marian Czarnecki Tom Schaul Joel Z Leibo David Silver Koray Kavukcuoglu
references.title: Reinforcement learning with unsupervised auxiliary tasks
references.venue: arXiv preprint arXiv:1611.05397,
references.authors: Rie Johnson Tong Zhang
references.title: Accelerating stochastic gradient descent using predictive variance reduction
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Leslie Pack Kaelbling
references.title: Learning to achieve goals
references.venue: In IJCAI,
references.authors: Sham Kakade John Langford
references.title: Approximately optimal approximate reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Bilal Kartal Pablo Hernandez-Leal Matthew E Taylor
references.title: Terminal prediction as an auxiliary task for deep reinforcement learning
references.venue: In AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment,
references.authors: Timothy P Lillicrap Jonathan J Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver Daan Wierstra
references.title: Continuous control with deep reinforcement learning
references.venue: In International Conference on Learning Representations,
references.authors: Kaixiang Lin Jiayu Zhou
references.title: Ranking policy gradient
references.venue: In International Conference on Learning Representations,
references.authors: Hao Liu Yihao Feng Yi Mao Dengyong Zhou Jian Peng Qiang Liu
references.title: Action-dependent control variates for policy optimization via stein identity
references.venue: In International Conference on Learning Representations,
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra Martin Riedmiller
references.title: Playing atari with deep reinforcement learning
references.venue: arXiv preprint arXiv:1312.5602,
references.authors: Volodymyr Mnih Adria Puigdomenech Badia Mehdi Mirza Alex Graves Timothy Lillicrap Tim Harley David Silver Koray Kavukcuoglu
references.title: Asynchronous methods for deep reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: H. Namkoong J.C. Duchi
references.title: Variance-based regularization with convex objectives
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jan Peters Katharina Mulling Yasemin Altun
references.title: Relative entropy policy search
references.venue: In AAAI Conference on Artificial Intelligence,
references.authors: T Pham-Gia TL Hung
references.title: The mean and median absolute deviations
references.venue: Mathematical and Computer Modelling,
references.authors: M. Puterman
references.title: Markov Decision Processes: Discrete Stochastic Dynamic Programming
references.authors: Jürgen Schmidhuber
references.title: Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts
references.venue: Connection Science,
references.authors: J. Schulman S. Levine P. Abbeel M. Jordan P. Moritz
references.title: Trust region policy optimization
references.venue: In International Conference on Machine Learning,
references.authors: J. Schulman F. Wolski P. Dhariwal A. Radford O. Klimov
references.title: Proximal policy optimization algorithms
references.venue: arXiv preprint arXiv:1707.06347,
references.authors: John Schulman Philipp Moritz Sergey Levine Michael Jordan Pieter Abbeel
references.title: High-dimensional continuous control using generalized advantage estimation
references.venue: In International Conference on Learning Representations,
references.authors: David Silver Guy Lever Nicolas Heess Thomas Degris Daan Wierstra Martin Riedmiller
references.title: Deterministic policy gradient algorithms
references.venue: In International Conference on Machine Learning,
references.authors: Richard S Sutton David A McAllester Satinder P Singh Yishay Mansour
references.title: Policy gradient methods for reinforcement learning with function approximation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: P. Thodoroff A. Durand J. Pineau D. Precup
references.title: Temporal regularization for markov decision process
references.venue: In Advances in Neural Information Processing Systems,
references.authors: E. Todorov T. Erez Y. Tassa
references.title: Mujoco: A physics engine for model-based control
references.venue: In IEEE/RSJ International Conference on Intelligent Robots and Systems,
references.authors: Michel Tokic
references.title: Adaptive ε-greedy exploration in reinforcement learning based on value differences
references.venue: In Annual Conference on Artificial Intelligence,
references.authors: George Tucker Surya Bhupatiraju Shixiang Gu Richard Turner Zoubin Ghahramani Sergey Levine
references.title: The mirage of action-dependent baselines in reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Hado Van Hasselt Arthur Guez David Silver
references.title: Deep reinforcement learning with double qlearning
references.venue: arXiv preprint arXiv:1509.06461,
references.authors: Ziyu Wang Tom Schaul Matteo Hessel Hado Hasselt Marc Lanctot Nando Freitas
references.title: Dueling network architectures for deep reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: R.J. Williams
references.title: Simple statistical gradient-following algorithms for connectionist reinforcement learning
references.venue: Machine Learning,
references.authors: Ronald J Williams Jing Peng
references.title: Function optimization using connectionist reinforcement learning algorithms
references.venue: Connection Science,
references.authors: Cathy Wu Aravind Rajeswaran Yan Duan Vikash Kumar Alexandre M Bayen Sham Kakade Igor Mordatch Pieter Abbeel
references.title: Variance reduction for policy gradient with action-dependent factorized baselines
references.venue: In International Conference on Learning Representations,
references.authors: Tingting Zhao Gang Niu Ning Xie Jucheng Yang Masashi Sugiyama
references.title: Regularized policy gradients: direct variance reduction in policy gradient estimation
references.venue: In Asian Conference on Machine Learning,
references.authors: Ilyas
references.title: 2020) indicates that commonly used algorithms fail to fit the true value function. However, this does not rule out the usefulness of the approach but rather begs for more questioning of the true effect of such biased baselines
sections.heading: 1 INTRODUCTION
sections.text: Model-free deep reinforcement learning (RL) has been successfully used in a wide range of problem domains, ranging from teaching computers to control robots to playing sophisticated strategy games (Silver et al., 2014; Schulman et al., 2016; Lillicrap et al., 2016; Mnih et al., 2016). Stateof-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework (Sutton et al., 2000; Schulman et al., 2017; Haarnoja et al., 2018). While such methods demonstrate great performance in continuous control tasks, several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains.
  For instance, research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions, then propose a critic formulation that allows for a more stable policy gradient. However, new studies (Tucker et al., 2018; Ilyas et al., 2020) indicate that state-of-the-art policy gradient methods (Schulman et al., 2015; 2017) fail to fit the true value function and that recently proposed state-action-dependent baselines (Gu et al., 2016; Liu et al., 2018; Wu et al., 2018) do not reduce gradient variance more than state-dependent ones.
  These findings leave the reader skeptical about actor-critic algorithms, suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning. Consequently, attempting to find a better baseline is questionable, as critics would typically fail to fit it (Ilyas et al., 2020). In Tucker et al. (2018), the authors argue that “much larger gains could be achieved by instead improving the accuracy of the value function”. Following this line of thought, we are interested in ways to better approximate the value function. One approach addressing this issue is to put more focus on relative state-action values, an idea introduced in the literature on advantage reinforcement
  ∗Equal contribution.
  learning (Harmon & Baird III) followed by works on dueling (Wang et al., 2016) neural networks. More recent work (Lin & Zhou, 2020) also suggests that considering the relative action values, or more precisely the ranking of actions in a state leads to better policies. The main argument behind this intuition is that it suffices to identify the optimal actions to solve a task. We extend this principle of relative action value with respect to the mean value to cover both state and state-action-value functions with a new objective for the critic: minimizing the variance of residual errors.
  In essence, this modified loss function puts more focus on the values of states (resp. state-actions) relative to their mean value rather than their absolute values, with the intuition that solving a task corresponds to identifying the optimal action(s) rather than estimating the exact value of each state. In summary, this paper:
  • Introduces Actor with Variance Estimated Critic (AVEC), an actor-critic method providing a new training objective for the critic based on the residual variance.
  • Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modified gradient estimator.
  • Demonstrates experimentally that AVEC, when coupled with state-of-the-art policy gradient algorithms, yields a significant performance boost on a set of challenging tasks, including environments with sparse rewards.
  • Provides empirical evidence supporting a better fit of the true value function and a substantial stabilization of the gradient.
sections.heading: 2 RELATED WORK
sections.text: Our approach builds on three lines of research, of which we give a quick overview: policy gradient algorithms, regularization in policy gradient methods, and exploration in RL.
  Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator. This was originally formulated as the REINFORCE algorithm (Williams, 1992). Kakade & Langford (2002) later created conservative policy iteration and provided lower bounds for the minimum objective improvement. Peters et al. (2010) replaced regularization by a trust region constraint to stabilize training. In addition, extensive research investigated methods to improve the stability of gradient updates, and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories, the corresponding variance can be extremely high. To improve stability, Weaver & Tao (2001) show that subtracting a baseline (Williams, 1992) from the value function in the policy gradient can be very beneficial in reducing variance without damaging the bias. However, in practice, these modifications on the actor-critic framework usually result in improved performance without a significant variance reduction (Tucker et al., 2018; Ilyas et al., 2020). Currently, one of the most dominant on-policy methods are proximal policy optimization (PPO) (Schulman et al., 2017) and trust region policy optimization (TRPO) (Schulman et al., 2015), both of which require new samples to be collected for each gradient step. Another direction of research that overcomes this limitation is off-policy algorithms, which therefore benefit from all sample transitions; soft actor-critic (SAC) (Haarnoja et al., 2018) is one such approach achieving state-of-the-art performance.
  Several works also investigate regularization effects on the policy gradient (Jaderberg et al., 2016; Namkoong & Duchi, 2017; Kartal et al., 2019; Flet-Berliac & Preux, 2019; 2020); it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias. In RL, regularization is often used to encourage exploration and takes the form of an entropy term (Williams & Peng, 1991; Schulman et al., 2017). Moreover, while regularization in machine learning generally consists in smoothing over the observation space, in the RL setting, Thodoroff et al. (2018) show that it is possible to smooth over the temporal dimension as well. Furthermore, Zhao et al. (2016) analyze the effects of a regularization using the variance of the policy gradient (the idea is reminiscent of SVRG descent (Johnson & Zhang, 2013)) which proves to provide more consistent policy improvements at the expense of reduced performance. In contrast, as we will see later, AVEC does not change the policy network optimization procedure nor involves any additional computational cost.
  Exploration has been studied under different angles in RL, one common strategy is -greedy, where the agent explores with probability by taking a random action. This method, just like entropy regularization, enforces uniform exploration and has achieved recent success in game playing en-
  vironments (Mnih et al., 2013; Van Hasselt et al., 2015; Mnih et al., 2016). On the other hand, for most policy-based RL, exploration is a natural component of any algorithm following a stochastic policy, choosing sub-optimal actions with non-zero probability. Furthermore, policy gradient literature contains exploration methods based on uncertainty estimates of values (Kaelbling, 1993; Tokic, 2010), and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration (Schmidhuber, 2006; Bellemare et al., 2016; Flet-Berliac et al., 2021).
  While existing research may share some motivations with our method, no previous work in RL applies the variance of residual errors as an objective loss function. In the context of linear regression, Brown (1947) considers a median-unbiased estimator minimizing the risk with respect to the absolutedeviation loss function (Pham-Gia & Hung, 2001) (similar in spirit to the variance of residual errors), their motivation is nonetheless different to ours. Indeed, they seek to be robust to outliers whereas, when considering noiseless RL problems, one usually seeks to capture those (sometimes rare) signals corresponding to the rewards.
sections.heading: 3 PRELIMINARIES
sections.heading: 3.1 BACKGROUND AND NOTATIONS
sections.text: We consider an infinite-horizon Markov Decision Problem (MDP) with continuous states s ∈ S, continuous actions a ∈ A, transition distribution st+1 ∼ P(st, at) and reward function rt ∼ R(st, at). Let πθ(a|s) denote a stochastic policy with parameter θ, we restrict policies to being Gaussian distributions. In the following, π and πθ denote the same object. The agent repeatedly interacts with the environment by sampling action at ∼ π(.|st), receives reward rt and transitions to a new state st+1. The objective is to maximize the expected sum of discounted rewards:
  J(π) , Eτ∼π [ ∞∑ t=0 γtr (st, at) ] , (1)
  where γ ∈ [0, 1) is a discount factor (Puterman, 1994), and τ = (s0, a0, r0, s1, a1, r1, . . . ) is a trajectory sampled from the environment using policy π. We denote the value of a state s in the MDP framework while following a policy π by V π(s) , Eτ∼π [ ∑∞ t=0 γ
  tr (st, at) |s0 = s] and the value of a state-action pair of performing action a in state s and then following policy π by Qπ(s, a) , Eτ∼π [ ∑∞ t=0 γ
  tr (st, at) |s0 = s, a0 = a]. Finally, the advantage function which quantifies how an action a is better than the average action in state s is denoted Aπ(s, a) , Qπ(s, a)− V π(s).
sections.heading: 3.2 CRITICS IN DEEP POLICY GRADIENTS
sections.text: In this section, we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient. Without loss of generality, we consider the algorithms that approximate the state-value function V . The analysis holds for algorithms that approximate the state-action-value function Q. Let fφ : S → R be an estimator of V̂ π with φ its parameter. fφ is traditionally learned through minimizing the mean squared error (MSE) against V̂ π . At iteration k, the critic minimizes:
  LAC = Es [( fφ(s)− V̂ πθk (s) )2] , (2)
  where the states s are collected under policy πθk , and V̂ πθk (s) is an empirical estimate of V (see Section 4.3 for details). Similarly, using fφ : S ×A → R instead, one can fit an empirical target Q̂π .
sections.heading: 4 METHOD: ACTOR WITH VARIANCE ESTIMATED CRITIC
sections.text: In this section, we introduce AVEC and discuss its correctness, motivations and implementation.
sections.heading: 4.1 DEFINING AN ALTERNATIVE CRITIC
sections.text: Recent work (Ilyas et al., 2020) empirically demonstrates that while the value network succeeds in the supervised learning task of fitting V̂ π (resp. Q̂π), it does not fit V π (resp. Qπ). We address this
  deficiency in the estimation of the critic by introducing an alternative value network loss. Following empirical evidence indicating that the problem is the approximation error and not the estimator per se, AVEC adopts a loss that can provide a better approximation error, and yields better estimators of the value function (as will be shown in Section 5.3). At update k:
  LAVEC = Es [( (fφ(s)− V̂ πθk (s))− Es [ fφ(s)− V̂ πθk (s) ])2] , (3)
  with states s collected using πθk . Note that the gradient flows in fφ twice using Eq. 3. Then, we define our bias-corrected estimator: gφ : S → R such that gφ(s) = fφ(s) + Es[V̂ πθk (s) − fφ(s)]. Analogously to Eq. 3, we define an alternative critic for the estimation of Qπ by replacing V̂ π by Q̂π and fφ(s) by fφ(s, a).
  Proposition (AVEC Policy Gradient). If fφ : S × A → R satisfies the parameterization assumption (Sutton et al., 2000) then gφ provides an unbiased policy gradient:
  ∇θJ (πθ) = E(s,a)∼πθ [∇θ log(πθ(s, a))gφ(s, a)] .
  Proof. See Appendix A. This result also holds for the estimation of V πθ with fφ : S → R.
sections.heading: 4.2 BUILDING MOTIVATION
sections.text: Here, we present the intuition behind using AVEC for actor-critic algorithms. Tucker et al. (2018) and Ilyas et al. (2020) indicate that the approximation error ‖V̂ π − V π‖ is problematic, suggesting that the variance of the empirical targets V̂ π(st) is high. Using LAVEC, our approach reduces the variance term of the MSE (or distance to V π) but mechanistically also increases the bias. Our intuition is that since the bias is already quite substantial (Ilyas et al., 2020), it may be possible to reduce the variance enough so that even though the bias increases, the total MSE reduces.
  State-value function estimation. In this case, optimizing the critic with LAVEC can be interpreted as fitting V̂ ′π(s) = V̂ π(s) − Es′ [V̂ π(s′)] using the MSE. We show that the targets V̂ ′π are better estimations of V ′π(s) = V π(s) − Es′ [V π(s′)] than V̂ π are of V π. To illustrate this, consider T independent random variables (Xi)i∈{1,...,T}. We denote X ′i = Xi − 1T ∑T j=1Xj and V(X) the variance of X . Then, V (X ′i) = V (Xi) − 2T V (Xi) + 1 T 2 ∑T j=1 V (Xj) and V(X ′i) < V(Xi)
  as long as ∀i 1T ∑T j=1 V(Xj) < 2V(Xi), or more generally when state-values are not strongly negatively correlated1 and not very discordant. This entails that V̂ ′π has a more compact span, and is consequently easier to fit. This analysis shows that the variance term of the MSE is reduced compared to traditional actor-critic algorithms, but does not guarantee it counterbalances the bias increase. Nevertheless, in practice, the bias is so high that the difference due to learning with AVEC is only marginal and the total MSE decreases. We empirically demonstrate this claim in Section 5.3.
  0 2 4 6 8 0.0
  0.5
  1.0
  1.5
  2.0
  2.5
  3.0
  3.5 MSE
  Var Initial
  Figure 1: Comparison of simple models derived when LAVEC is used instead of the MSE.
  State-action-value function estimation. In this case, Eq. 3 translates into replacing V̂ π(s) by Q̂π(s, a) and fφ(s) by fφ(s, a) and the rationale for optimizing the residual variance of the value function instead of the full MSE becomes more straightforward: the practical use of the Q-function is to disentangle the relative values of actions for each state (Sutton et al., 2000). AVEC’s effect on relative values is illustrated in a didactic regression with one variable example in Fig. 1 where grey markers are observations and the blue line is our current estimation. Minimizing the MSE, the line is expected to move
  towards the orange one in order to reduce errors uniformly. Minimizing the residual variance, it
  1Greensmith et al. (2004) analyze the dependent case: in general, weakly dependent variables tend to concentrate more than independent ones.
  is expected to move near the red one. In fact, LAVEC tends to further penalize observations that are far away from the mean, implying that AVEC allows a better recovery of the “shape” of the target near extrema. In particular, we see in the figure that the maximum and minimum observation values are quickly identified. Would the approximators be linear and the target state-values independent, the two losses become equivalent since ordinary least squares would provide minimum-variance mean-unbiased estimation.
  It should be noted that, as in all the works related to ours, we consider noiseless tasks, i.e. the transition matrix is deterministic. As such, there are no outliers and extreme state-action values correspond to learning signals. In this context, high estimation errors indicate where (in the state or action-state space) the training of the value function should be improved.
sections.heading: 4.3 IMPLEMENTATION
sections.text: We apply this new formulation to three of the most dominant deep policy gradient methods to study whether it results in a better estimation of the value function. A better estimation of the value function implies better policy improvements. We now describe how AVEC incorporates its residual variance objective into the critics of PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015) and SAC (Haarnoja et al., 2018). Let B be a batch of transitions. In PPO and TRPO, AVEC modifies the learning of Vφ (line 12 of Algorithm 1) using:
  L1AVEC (φ) = Es∼B [ (fφ (s)− V̂ π (s))− Es∼B [ fφ (s)− V̂ π (s) ] ]2 ,
  then Vφ = fφ(s) + Es∼B[V̂ π(s) − fφ(s)], where V̂ π (st) = fφold(st) + At such that fφold(st) are the estimates given by the last value function and At is the advantage of the policy, i.e. the returns minus the expected values (At is often estimated using generalized advantage estimation (Schulman et al., 2016). In SAC, AVEC modifies the objective function of (Qφi)i=1,2 (line 13 of Algorithm 2 in Appendix C) using:
  L2AVEC (φi) = E(s,a)∼B [ (fφi(s, a)− Q̂π(s, a))− E(s,a)∼B [ fφi(s, a)− Q̂π(s, a) ] ]2 ,
  then Qφi = fφi(s, a) + E(s,a)∼B[Q̂π(s, a)− fφi(s, a)], where Q̂π(s, a) is estimated using temporal difference (see Haarnoja et al. (2018)): Q̂π(st, at) = r(st, at) + γEst+1∼π[Vψ̄(st+1)] with ψ̄ the value function parameter (see Algorithm 2). The reader may have noticed that L1AVEC and L2AVEC slightly differ from Eq. 3. The residual variance of the value function (LAVEC) is not tractable since a priori state-values are dependent and their joint law is unknown. Consequently, in practice, we use the empirical variance proxy assuming independence (cf. Appendix D). Greensmith et al. (2004) provide some support for this approximation by showing that weakly dependent variables tend to concentrate more than independent ones. Finally, notice that AVEC does not modify any other part of the considered algorithms whatsoever, which makes its implementation straightforward and keeps the same computational complexity.
sections.heading: 5 EXPERIMENTAL STUDY
sections.text: In this section, we conduct experiments along four orthogonal directions. (a) We validate the superiority of AVEC compared to the traditional actor-critic training. (b) We evaluate AVEC in environments with sparse rewards. (c) We clarify the practical implications of using AVEC by examining the bias in both the empirical and true value function estimations as well as the variance in the empirical gradient. (d) We provide an ablation analysis and study the bias-variance trade-off in the critic by considering two continuous control tasks. We point out that a comparison to variance-reduction methods is not considered in this paper: Tucker et al. (2018) demonstrated that their implementations diverge from the unbiased methods presented in the respective papers and unveiled that not only do they fail to reduce the variance of the gradient, but that their unbiased versions do not improve performance either. Note that in all experiments we choose the hyperparameters providing the best performance for the considered methods which can only penalyze AVEC (cf. Appendix E). In all the figures hereafter (except Fig. 3c and 3d), lines are average performances and shaded areas represent one standard deviation.
  5.1 CONTINUOUS CONTROL
  Algorithm 1 AVEC coupled with PPO or TRPO. JALGO denotes the policy loss of either algorithm (described in Schulman et al. (2017; 2015)).
  1: Input parameters: λπ ≥ 0, λV ≥ 0 2: Initialize policy parameter θ and value func-
  tion parameter φ 3: for each update step do 4: batch B ← ∅ 5: for each environment step do 6: at ∼ πθ(st) 7: st+1 ∼ P (st, at) 8: B ← B ∪ {(st, at, rt, st+1)} 9: end for
  10: for each gradient step do 11: θ ← θ − λπ∇̂θJALGO(πθ) 12: φ← φ− λV ∇̂φL1AVEC (φ) 13: end for 14: end for For ease of comparison with other methods, we evaluate AVEC on the MuJoCo (Todorov et al., 2012) and the PyBullet (Coumans & Bai, 2016) continuous control benchmarks (see Appendix G for details) using OpenAI Gym (Brockman et al., 2016). Note that the PyBullet versions of the locomotion tasks are harder than the MuJoCo equivalents2. We choose a representative set of tasks for the experimental evaluation; their action and observation space dimensions are reported in Appendix H. We assess the benefits of AVEC when coupled with the most prominent policy gradient algorithms, currently state-of-the-art methods: PPO (Schulman et al., 2017) and TRPO (Schulman et al., 2015), both on-policy methods, and SAC (Haarnoja et al., 2018), an off-policy maximum entropy deep RL algorithm. We provide the list of hyperparameters and further implementation details in Appendix D and E.
  Table 1 reports the results while Fig. 2 and 8 show the total average return for SAC and PPO. TRPO results are provided in Appendix F for readability. When coupled with SAC and PPO, AVEC brings very significant improvement (on average +26% for SAC and +39% for PPO) in the performance of the policy gradient algorithms, improvement which is consistent across tasks. As for TRPO, while the improvement in performance is less striking, AVEC still manages to be more efficient in terms of sampling in all tasks. Overall, AVEC improves TRPO, PPO and SAC in terms of performance and efficiency. This does not imply that our method would also improve other policy gradient methods that use the traditional actor-critic framework, but since we evaluate our method coupled with three of the best performing on- and off-policy algorithms, we believe that these experiments are sufficient to prove the relevance of AVEC. Furthermore, in our experiments we do not seek the best hyperparameters for the AVEC variants, we simply adopt the parameters allowing us to optimally reproduce the baselines. Alternatively, if one seeks to evaluate AVEC independently of a considered baseline, further hyperparameter tuning should produce better results. Notice that since no additional calculations are needed in AVEC’s implementation, computational complexity remains unchanged.
sections.heading: 5.2 SPARSE REWARD SIGNALS
sections.text: Domains with sparse rewards are challenging to solve with uniform exploration as agents receive no feedback on their actions before starting to collect rewards. In such conditions AVEC performs better, suggesting that the shape of the value function is better approximated, encouraging exploration.
  2Bullet Physics SDK GitHub Issue.
  The relative value estimate of an unseen state is more accurate: in Section 4.2, AVEC identifies extreme state-values (e.g., non-zero rewards in tasks with sparse rewards) faster. In Fig. 3a and 3b, we report the performance of AVEC in the Acrobot and MountainCar environments: both have sparse rewards. AVEC enhances TRPO and PPO in both experiments. When PPO and AVEC-PPO both reach the best possible performance, AVEC-PPO exhibits better sample efficiency. Fig. 3c and 3d illustrate how the agent improves its exploration strategy in MountainCar: while the PPO agent remains stuck at the bottom of the hill (red), the graph suggest that AVEC-PPO learns the difficult locomotion principles in the absence of rewards and visits a much larger part of the state space (green).
  This improved performance in sparse environments can be explained by the fact that AVEC is able to pick up on experienced positive reward more easily. Moreover, the reconstructed shape of the value function is more accurate around such rewarding states, which pushes the agent to explore further around experienced states with high values.
sections.heading: 5.3 ANALYSIS OF THE VARIANCE ESTIMATED CRITIC
sections.text: In order to further validate AVEC, we evaluate the performance of the value network in more detail: we examine (a) the estimation error (distance to the empirical target), (b) the approximation error (distance to the true target) and (c) the empirical variance of the gradient. (a,b) should be put into perspective with the conclusions of Ilyas et al. (2020) where it is found that the critic only fits the empirical value function but not the true one. (c) should be placed in light of Tucker et al. (2018) highlighting a failure of recently proposed state-action-dependent baselines to reduce the variance.
  Learning the Empirical Target. In Fig. 4, we report the quality of fit (MSE) of the empirical target V̂ π in the methods PPO and AVEC-PPO in the AntBullet and HalfCheetahBullet tasks.
  0.0 0.2 0.4 0.6 0.8 1.0 Timesteps 1e6
  0
  100
  200
  300
  400
  D is
  ta nc
  e to
  e m
  pi ric
  al V
  -fu nc
  tio n
  AntBullet AVEC-PPO PPO
  0.0 0.2 0.4 0.6 0.8 1.0 Timesteps 1e6
  100
  200
  300
  400
  500
  D is
  ta nc
  e to
  e m
  pi ric
  al V
  -fu nc
  tio n
  HalfCheetahBullet AVEC-PPO PPO
  Figure 4: L2 distance to V̂ π .
  We observe that PPO better fits the empirical target than when equipped with AVEC, which is to be expected since vanilla PPO optimizes the MSE directly. This result put aside the remarkable improvement in the performance of AVEC-PPO (Fig. 2) suggests that AVEC might be a better estimator of the true value function. We examine this claim below because if true, it
  would indicate that it is indeed possible to simultaneously improve the performance of the agents and the stability of the method.
  Learning the True Target. A fundamental premise of policy gradient methods is that optimizing the objective based on an empirical estimation of the value function leads to a better policy. Which is why we investigate the quality of fit of the true target. To approximate the true value function, we fit the returns sampled from the current policy using a large number of transitions (3 · 105). Fig. 5 shows that gφ is far closer to the true value function half of the time (horizon is 106) than the estimator obtained
  with MSE, then as close to it. Comparing Fig. 5 with Fig. 4, we see that the distance to the true target is close to the estimation error for AVEC-PPO, while for PPO, it is at least two orders of magnitude higher at all times. We further investigate these results in Fig. 9 in Appendix B.2 where we study the variation of the squared bias and variance components of the MSE to the true target (MSE = Var + Bias2). We find, as expected, that using AVEC reduces the variance term significantly while slightly increasing the bias term, which Fig. 5 confirms is negligible since the total MSE is substantially reduced (‖gφ(AVEC)− V π‖2 ≤ ‖Vφ(PPO)− V π‖2) where Vφ(PPO) is the value function estimator in PPO. For completeness, we also analyze the distance to the true target for the Q-function estimator in SAC and AVEC-SAC in AntBullet and HalfCheetahBullet in Appendix B.3, with similar results and interpretation. We conclude that AVEC improves the value function approximation and we expect that the gradient is more stable.
  Empirical Variance Reduction. We choose to study the gradient variance using the average pairwise cosine similarity metric as it allows a comparison with Ilyas et al. (2020), with which we share the same experimental setup and scales. Fig. 6 shows that AVEC yields a higher average (10 batches per iteration) pairwise cosine similarity, which means closer batch-estimates of
  the gradient and, in turn, indicates smaller gradient variance. Further analysis with additional tasks is included in Appendix B.4. The variance reduction effect observed in several environments suggests that AVEC is the first method since the introduction of the value function baseline to further reduce the variance of the gradient and improve performance.
sections.heading: 5.4 ABLATION STUDY
sections.text: In this section, we examine how changing the relative importance of the bias and the residual variance in the loss of the value network affects learning. For this study, we choose difficult tasks of PyBullet and use PPO because it is more efficient than TRPO and requires less computations than SAC. For an estimator ŷn of (yi)i∈{1,...,n}, we write Bias = 1n ∑n i=1(ŷi − yi) and Var =
  1 n−1 ∑n i=1(ŷi − yi −Bias)2. Consequently: MSE = Var + Bias
  2. We denote Lα = Var + αBias2, with α ∈ R. In Fig. 7, Bias-α means that we use Lα and Var-α means that we use L 1
  α . We observe
  that while no consistent order on the choices of α is identified, AVEC seems to outperform all other weightings. Note that, for readability purposes, the graphs have been split and the curves of AVEC-PPO and PPO are the same in Fig. 7a and 7c, and in Fig. 7b and 7d. A more extensive hyperparameter
  study with more α values might provide even higher performances, nevertheless we believe that the stability of an algorithm is crucial for a reliable performance. As such, the tuning of hyperparameters to achieve good results should remain mild.
sections.heading: 6 DISCUSSION
sections.text: In this work, we introduce a new training objective for the critic in actor-critic algorithms to better approximate the true value function. In addition to being well-motivated by recent studies on the behaviour of deep policy gradient algorithms, we demonstrate that this modification is both theoretically sound and intuitively supported by the need to improve the approximation error of the critic. The application of Actor with Variance Estimated Critic (AVEC) to state-of-the-art policy gradient methods produces considerable gains in performance (on average +26% for SAC and +39% for PPO) over the standard actor-critic training, without any additional hyperparameter tuning.
  First, for SAC-like algorithms where the critic learns a state-action-value function, our results strongly suggest that state-actions with extreme values are identified more quickly. Second, for PPO-like methods where the critic learns the state-values, we show that the variance of the gradient is reduced and empirically demonstrate that this is due to a better approximation of the state-values. In sparse reward environments, the theoretical intuition behind a variance estimated critic is more explicit and is also supported by empirical evidence. In addition to corroborating the results in Ilyas et al. (2020) proving that the value estimator fails to fit V π, we propose a method that succeeds in improving both the sample complexity and the stability of prominent actor-critic algorithms. Furthermore, AVEC benefits from its simplicity of implementation since no further assumptions are required (such as horizon awareness Tucker et al. (2018) to remedy the deficiency of existing variance-reduction methods) and the modification of current algorithms represents only a few lines of code.
  In this paper, we have demonstrated the benefits of a more thorough analysis of the critic objective in policy gradient methods. Despite our strongly favourable results, we do not claim that the residual variance is the optimal loss for the state-value or the state-action-value functions, and we note that the design of comparably superior estimators for critics in deep policy gradient methods merits further study. In future work, further analysis of the bias-variance trade-off and extension of the results to stochastic environments is anticipated; we consider the problem of noise separation in the latter, as this is the first obstacle to accessing the variance and distinguishing extreme values from outliers.
sections.heading: B ADDITIONAL EXPERIMENTS
sections.text: B.1 CONTINUOUS CONTROL: WALKER2D
  Fig. 8 shows the total average return for AVEC coupled with SAC and PPO on the Walker2d task. Similar to considered other continuous control tasks from MuJoCo and PyBullet, AVEC brings a significant performance improvement (+26% for SAC and +33% for PPO), confirming the generality of our approach.
  B.2 VARIATION OF THE BIAS AND VARIANCE TERMS: PPO
  In Fig. 9, we show the variation of the bias and variance terms in the MSE between the estimators (of AVEC-PPO and PPO) and the true target: E[‖gφ − V π‖22] = Bias(AVEC)2 + Var(AVEC) and E[‖Vφ(PPO)− V π‖22] = Bias(PPO)2 + Var(PPO) where Vφ(PPO) is the value function estimator in PPO. We observe that the variance reduction is more substantial than that of the bias. Using those results and Fig. 5 showing that the distance of the estimator to V π is lower when using AVEC confirms that the variance reduction effect counterbalances the bias increase. Note that the % Variation of the Var term is always negative in our experiments, and that the shaded areas that suggest otherwise are merely due to a false assumption of symmetrical deviations, itself due to the assumption of Gaussianity needed to construct confidence intervals.
  target: %Variation(Bias) = Bias
  2(AVEC-PPO)−Bias2(PPO) Bias2(PPO) and %Variation(Var) = Var(AVEC-PPO)−Var(PPO) Var(PPO) .
  X-axis: we run PPO and AVEC-PPO and for every t ∈ {1, 2, 4, 6, 9} · 105, we stop training, use the current policy to interact with the environment for 3 · 105 transitions, and use these transitions to estimate the true value function. Lines are average variations and shaded areas represent one standard deviation (5 seeds).
  B.3 LEARNING THE TRUE TARGET: SAC
  In Fig. 10, we compare the error between the Q-function estimator and the true Q-function for SAC and AVEC-SAC in AntBullet and HalfCheetahBullet. We note a modest but consistent reduction in this error when using AVEC coupled with SAC, echoing the significant performance gains in Fig. 2.
  B.4 VARIANCE REDUCTION
  In Fig. 11, we study the empirical variance of the gradient in measuring the average pairwise cosine similarity (10 gradient measurements) in two additional tasks: HopperBullet and Walker2DBullet. We also vary the trajectory size used in the estimation of the gradient.
  C IMPLEMENTATION OF AVEC COUPLED WITH SAC
  In Algorithm 2, JV is the squared residual error objective to train the soft value function. See Haarnoja et al. (2018) for further details and notations about SAC, not directly relevant here.
  Algorithm 2 AVEC coupled with SAC. 1: Input parameters: β ∈ [0, 1], λV ≥ 0, λQ ≥ 0, λπ ≥ 0 2: Initialize policy parameter θ, value function parameter ψ and ψ̄ and Q-functions parameters φ1
  and φ2 3: D ← ∅ 4: for each iteration do 5: for each step do 6: at ∼ πθ(at|st) 7: st+1 ∼ P (st, at) 8: D ← D ∪ {(st, at, rt, st+1)} 9: end for
  10: for each gradient step do 11: sample batch B from D
sections.heading: 12: ψ ← ψ − λV ∇̂ψJV (ψ)
sections.text: 13: φi ← φi − λQ∇̂φiL2AVEC (φi) for i ∈ {1, 2} 14: θ ← θ − λπ∇̂θJ(πθ) 15: ψ̄ ← βψ + (1− β)ψ̄ 16: end for 17: end for
  D IMPLEMENTATION DETAILS
  Theoretically, LAVEC is defined as the residual variance of the value function (cf. Eq. 3). However, state-values for a non-optimal policy are dependent and the variance is not tractable without access to the joint law of state-values. Consequently, to implement AVEC in practice we use the best-known proxy at hand, which is the empirical variance formula assuming independence:
  LAVEC = 1
  T − 1 T∑ t=1 (( fφ(st)− V̂ π(st) ) − 1 T T∑ t=1 ( fφ(st)− V̂ π(st) ))2 ,
  where T is the size of the sampled trajectory.
sections.heading: E EXPERIMENT DETAILS
sections.text: In all experiments we choose to use the same hyperparameter values for all tasks as the bestperforming ones reported in the literature or in their respective open source implementation documentation. We thus ensure the best performance for the conventional actor-critic framework. In other words, since we are interested in evaluating the impact of this new critic, everything else is kept as is. This experimental protocol may not benefit AVEC.
  In Table 2, 3 and 4, we report the list of hyperparameters common to all continuous control experiments.
  F COMPARATIVE EVALUATION OF AVEC WITH TRPO
  In order to evaluate the performance gains in using AVEC instead of the usual actor-critic framework, we produce some additional experiments with the TRPO (Schulman et al., 2015) algorithm. Fig. 12 shows the learning curves while Table 5 reports the results.
sections.heading: H DIMENSIONS OF STUDIED TASKS
sections.heading: G ENVIRONMENTS DETAILS
title: LEARNING VALUE FUNCTIONS IN DEEP POLICY GRA-