summaries: This paper studies the problem of KL-regularized reinforcement learning for locomotion tasks, where the KL regularization penalizes deviations from expert demonstrations.  This work makes the observation that fitting expert demonstrations to a certain class of conditional neural density models (e.g. neural networks that output the parameters of a Gaussian distribution), results in policies whose variance collapses for states that are different enough from the expert data.   The paper argues that this collapse causes instabilities in learning KL-regularized policies: for Gaussian policies, the KL penalty grows quadratically to infinity as the expert policy variance goes to 0. To address this issue, the paper proposes to instead compute the KL penalty by fitting expert demonstrations to models that do not suffer from this collapse, in particular Gaussian Process regression models. Gaussian Process models result in variance that increases (depending on the choice of kernel) for states that are far enough from the expert data.  The paper provides some analysis on why the variance collapse affects the RL optimization process, experiments that  aim to show an empirical relationship between expert policy variance, KL penalty and magnitude of the policy gradients, and a comparison between various methods that use KL regularization with an expert policy.  The authors identify a problem with KL-based reinforcement learning from expert demonstrations. Poor out-of-sample variance predictions of NN expert policies ends up unreasonably penalizing RL exploration outside the expert data distribution, potentially causing numerical instability. To remedy this they propose to instead use non-parametric expert policies based on a scalable GP approach.  This paper presents a previously unrecognized pathology in KL-regularized RL with expert demonstrations. Specifically the commonly used parametric behavioral policies suffer from collapse of predictive variance at states far from the demonstrations. This collapse of variance hinders the algorithm’s ability to learn effectively. The authors then propose to use non-parametric behavioral policies and demonstrate its effectiveness in several control tasks.  The authors identify a potential failure in KL-regularized reinforcement learning in which small predictive variance of the target policy may lead to exploding gradients, possibly destabilizing gradient-based learning algorithms. The authors discuss an uncertainty collapse of parametric models under Maximum Likelihood estimation in OOD regions of the expert, resulting in the aforementioned pathology. Finally, they suggest to use non-parametric models of the behavior policy (in which uncertainty collapse is not present), showing improved performance on several continuous control benchmarks.
abstractText: KL-regularized reinforcement learning from expert demonstrations has proved successful in improving the sample efficiency of deep reinforcement learning algorithms, allowing them to be applied to challenging physical real-world tasks. However, we show that KL-regularized reinforcement learning with behavioral reference policies derived from expert demonstrations can suffer from pathological training dynamics that can lead to slow, unstable, and suboptimal online learning. We show empirically that the pathology occurs for commonly chosen behavioral policy classes and demonstrate its impact on sample efficiency and online policy performance. Finally, we show that the pathology can be remedied by non-parametric behavioral reference policies and that this allows KL-regularized reinforcement learning to significantly outperform state-of-the-art approaches on a variety of challenging locomotion and dexterous hand manipulation tasks.
authors.name: Tim G. J. Rudner
authors.name: Cong Lu
authors.name: Michael A. Osborne
references.authors: Michael Bain Claude Sammut
references.title: A framework for behavioural cloning
references.venue: In Machine Intelligence
references.authors: Philip J Ball Cong Lu Jack Parker-Holder Stephen Roberts
references.title: Augmented world models facilitate zero-shot dynamics generalization from a single offline environment
references.venue: Proceedings of the 38th International Conference on Machine Learning,
references.authors: Abdeslam Boularias Jens Kober Jan Peters
references.title: Relative entropy inverse reinforcement learning
references.venue: In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,
references.authors: Ivan Bratko Tanja Urbancic Claude Sammut
references.title: Behavioural cloning: phenomena, results and problems
references.venue: IFAC Proceedings Volumes,
references.authors: Tim Brys Anna Harutyunyan Halit Bener Suay Sonia Chernova Matthew E Taylor Ann Nowé
references.title: Reinforcement learning from demonstrations through shaping
references.venue: In Twenty-fourth International Joint Conference on Artificial Intelligence,
references.authors: Catherine Cang Aravind Rajeswaran Pieter Abbeel Michael Laskin
references.title: Behavioral priors and dynamics models: Improving performance and domain transfer in offline RL, 2021
references.authors: Thomas Degris Martha White Richard S. Sutton
references.title: Off-policy actor-critic
references.venue: In Proceedings of the 29th International Coference on International Conference on Machine Learning,
references.authors: Gabriel Dulac-Arnold Daniel Mankowitz Todd Hester
references.title: Challenges of real-world reinforcement learning
references.venue: arXiv preprint arXiv:1904.12901,
references.authors: Sebastian Farquhar Michael A. Osborne Yarin Gal
references.title: Radial bayesian neural networks: Beyond discrete support in large-scale bayesian deep learning
references.venue: Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics,
references.authors: Alexandre Galashov Siddhant M. Jayakumar Leonard Hasenclever Dhruva Tirumala Jonathan Schwarz Guillaume Desjardins Wojciech M. Czarnecki Yee Whye Teh Razvan Pascanu Nicolas Heess
references.title: Information asymmetry in kl-regularized RL
references.venue: In 7th International Conference on Learning Representations,
references.authors: Yang Gao Huazhe Xu Ji Lin Fisher Yu Sergey Levine Trevor Darrell
references.title: Reinforcement learning from imperfect demonstrations
references.venue: arXiv preprint arXiv:1802.05313,
references.authors: CW Groetsch
references.title: The theory of Tikhonov regularization for Fredholm equations
references.venue: Boston Pitman Publication,
references.authors: Tuomas Haarnoja Aurick Zhou Kristian Hartikainen George Tucker Sehoon Ha Jie Tan Vikash Kumar Henry Zhu Abhishek Gupta Pieter Abbeel Sergey Levine
references.title: Soft actor-critic algorithms and applications, 2019
references.authors: Leslie Pack Kaelbling Michael L Littman Andrew W Moore
references.title: Reinforcement learning: A survey
references.venue: Journal of artificial intelligence research,
references.authors: Rahul Kidambi Aravind Rajeswaran Praneeth Netrapalli Thorsten Joachims
references.title: MOReL: Model-based offline reinforcement learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Vijay R Konda John N Tsitsiklis
references.title: Actor-critic algorithms. In Advances in neural information processing systems, pages 1008–1014
references.authors: George Konidaris Scott Kuindersma Roderic Grupen Andrew Barto
references.title: Robot learning from demonstration by constructing skill trees
references.venue: The International Journal of Robotics Research,
references.authors: Aviral Kumar Justin Fu Matthew Soh George Tucker Sergey Levine
references.title: Stabilizing off-policy Q-learning via bootstrapping error reduction
references.venue: In Advances in Neural Information Processing Systems
references.authors: Balaji Lakshminarayanan Alexander Pritzel Charles Blundell
references.title: Simple and scalable predictive uncertainty estimation using deep ensembles
references.venue: Advances in Neural Information Processing Systems
references.authors: Sergey Levine
references.title: Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018
references.authors: Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver Daan Wierstra
references.title: Continuous control with deep reinforcement learning
references.venue: In ICLR (Poster),
references.authors: Cong Lu Philip J. Ball Jack Parker-Holder Michael A. Osborne Stephen J. Roberts
references.title: Revisiting design choices in model-based offline reinforcement learning, 2021
references.authors: Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra Martin A. Riedmiller
references.title: Playing atari with deep reinforcement learning
references.venue: CoRR, abs/1312.5602,
references.authors: Kevin P. Murphy
references.title: Machine learning: A Probabilistic Perspective
references.authors: A. Nair B. McGrew M. Andrychowicz W. Zaremba P. Abbeel
references.title: Overcoming exploration in reinforcement learning with demonstrations
references.venue: IEEE International Conference on Robotics and Automation (ICRA),
references.authors: Ashvin Nair Murtaza Dalal Abhishek Gupta Sergey Levine
references.title: Accelerating online reinforcement learning with offline datasets, 2020
references.authors: Nicolás Navarro-Guerrero Cornelius Weber Pascal Schroeter Stefan Wermter
references.title: Real-world reinforcement learning for autonomous humanoid robot docking
references.venue: Robotics and Autonomous Systems,
references.authors: Andrew Y. Ng Stuart J. Russell
references.title: Algorithms for inverse reinforcement learning
references.venue: In Proceedings of the Seventeenth International Conference on Machine Learning,
references.authors: Aldo Pacchiano Jack Parker-Holder Yunhao Tang Krzysztof Choromanski Anna Choromanska Michael Jordan
references.title: Learning to score behaviors for guided policy optimization
references.venue: Proceedings of the 37th International Conference on Machine Learning,
references.authors: Xue Bin Peng Aviral Kumar Grace Zhang Sergey Levine
references.title: Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019
references.authors: Jan Peters Sethu Vijayakumar Stefan Schaal
references.title: Natural actor-critic
references.venue: In European Conference on Machine Learning,
references.authors: Joaquin Quiñonero Candela Carl Edward Rasmussen
references.title: A unifying view of sparse approximate Gaussian process regression
references.venue: J. Mach. Learn. Res.,
references.authors: Aravind Rajeswaran Vikash Kumar Abhishek Gupta Giulia Vezzani John Schulman Emanuel Todorov Sergey Levine
references.title: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations
references.venue: In Proceedings of Robotics: Science and Systems (RSS),
references.authors: Aravind Rajeswaran Vikash Kumar Abhishek Gupta Giulia Vezzani John Schulman Emanuel Todorov Sergey Levine
references.title: Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018
references.authors: Carl Edward Rasmussen Christopher K.I. Williams
references.title: Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)
references.authors: Konrad Rawlik Marc Toussaint Sethu Vijayakumar
references.title: On stochastic optimal control and reinforcement learning by approximate inference
references.venue: Proceedings of Robotics: Science and Systems VIII,
references.authors: Michael T Rosenstein Andrew G Barto Jennie Si Andy Barto Warren Powell
references.title: Supervised actor-critic reinforcement learning. Learning and Approximate Dynamic Programming: Scaling
references.venue: Up to the Real World,
references.authors: Tim G.J. Rudner Zonghao Chen Yarin Gal
references.title: Rethinking function-space variational inference in Bayesian neural networks
references.venue: In Third Symposium on Advances in Approximate Bayesian Inference,
references.authors: Tim G.J. Rudner Vitchyr H. Pong Rowan McAllister Yarin Gal Sergey Levine
references.title: Outcomedriven reinforcement learning via variational inference
references.venue: In Advances in Neural Information Processing Systems
references.authors: Tim G.J. Rudner Freddie Bickford Smith Qixuan Feng Yee Whye Teh Yarin Gal
references.title: Continual learning via function-space variational inference
references.venue: In ICML Workshop on Theory and Foundations of Continual Learning
references.authors: Stefan Schaal
references.title: Learning from demonstration
references.venue: Advances in neural information processing systems,
references.authors: John Schulman Xi Chen Pieter Abbeel
references.title: Equivalence between policy gradients and soft Q-learning
references.venue: arXiv preprint arXiv:1704.06440,
references.authors: Noah Siegel Jost Tobias Springenberg Felix Berkenkamp Abbas Abdolmaleki Michael Neunert Thomas Lampe Roland Hafner Nicolas Heess Martin Riedmiller
references.title: Keep doing what worked: Behavior modelling priors for offline reinforcement learning
references.venue: In International Conference on Learning Representations,
references.authors: Alex Smola Arthur Gretton Le Song Bernhard Schölkopf
references.title: A hilbert space embedding for distributions
references.venue: Algorithmic Learning Theory,
references.authors: Richard S. Sutton Andrew G. Barto
references.title: Reinforcement Learning: An Introduction
references.authors: Gerald Tesauro
references.title: Temporal difference learning and td-gammon
references.venue: Communications of the ACM,
references.authors: Emanuel Todorov
references.title: Linearly-solvable markov decision problems
references.venue: Advances in Neural Information Processing Systems,
references.authors: Joost van Amersfoort Lewis Smith Andrew Jesson Oscar Key Yarin Gal
references.title: Improving deterministic uncertainty estimation in deep learning for classification and regression, 2021
references.authors: Ke Wang Geoff Pleiss Jacob Gardner Stephen Tyree Kilian Q Weinberger Andrew Gordon Wilson
references.title: Exact Gaussian processes on a million data points
references.venue: Advances in Neural Information Processing Systems,
references.authors: Yifan Wu George Tucker Ofir Nachum
references.title: Behavior regularized offline reinforcement learning
references.venue: arXiv preprint arXiv:1911.11361,
references.authors: Tianhe Yu Garrett Thomas Lantao Yu Stefano Ermon James Y Zou Sergey Levine Chelsea Finn Tengyu Ma
references.title: Mopo: Model-based offline policy optimization
references.venue: Advances in Neural Information Processing Systems,
references.authors: Tianhe Yu Aviral Kumar Rafael Rafailov Aravind Rajeswaran Sergey Levine Chelsea Finn
references.title: COMBO: Conservative offline model-based policy optimization, 2021
sections.heading: 1 Introduction
sections.text: Reinforcement learning (RL) [15, 24, 46, 47] is a powerful paradigm for learning complex behaviors. Unfortunately, many modern reinforcement learning algorithms require agents to carry out millions of interactions with their environment to learn desirable behaviors, making them of limited use for a wide range of practical applications that cannot be simulated [8, 28]. This limitation has motivated the study of algorithms that can incorporate pre-collected offline data into the training process either fully offline or with online exploration to improve sample efficiency, performance, and reliability [2, 6, 16, 23, 52, 53]. An important and well-motivated subset of these methods consists of approaches for efficiently incorporating expert demonstrations into the learning process [5, 11, 18, 42].
  Reinforcement learning with Kullback-Leibler (KL) regularization is a particularly successful approach for doing so [3, 27, 29, 31, 44, 51]. In KL-regularized reinforcement learning, the standard reinforcement learning objective is augmented by a Kullback-Leibler divergence term that penalizes dissimilarity between the online policy and a behavioral reference policy derived from expert demonstrations. The resulting regularized objective pulls the agent’s online policy towards the behavioral reference policy while also allowing it to improve upon the behavioral reference policy by exploring and interacting with the environment. Recent advances that leverage explicit or implicit KL-regularized objectives, such as BRAC [51], ABM [44], and AWAC [27], have shown that KLregularized reinforcement learning from expert demonstrations is able to significantly improve the sample efficiency of online training and reliably solve challenging environments previously unsolved by standard deep reinforcement learning algorithms.
  ∗Equal contribution. † Corresponding author: tim.rudner@cs.ox.ac.uk.
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  Contributions. In this paper, we show that despite some empirical success, KL-regularized reinforcement learning from expert demonstrations can suffer from previously unrecognized pathologies that lead to instability and sub-optimality in online learning. To summarize, our core contributions are as follows: • We illustrate empirically that commonly used classes of parametric behavioral policies experi-
  ence a collapse in predictive variance about states away from the expert demonstrations. • We demonstrate theoretically and empirically that KL-regularized reinforcement learning al-
  gorithms can suffer from pathological training dynamics in online learning when regularized against behavioral policies that exhibit such a collapse in predictive variance. • We show that the pathology can be remedied by non-parametric behavioral policies, whose
  predictive variances are well-calibrated and guaranteed not to collapse about previously unseen states, and that fixing the pathology results in online policies that significantly outperform stateof-the-art approaches on a range of challenging locomotion and dexterous hand manipulation tasks.
  The left panel of Figure 1 shows an example of the collapse in predictive variance away from the expert trajectories in parametric behavioral policies. In contrast, the right panel of Figure 1 shows the predictive variance of a non-parametric behavioral policy, which—unlike in the case of the parametric policy—increases off the expert trajectories. By avoiding the pathology, we obtain a stable and reliable approach to sample-efficient reinforcement learning, applicable to a wide range of reinforcement learning algorithms that leverage KL-regularized objectives.2
sections.heading: 2 Background
sections.text: We consider the standard reinforcement learning setting where an agent interacts with a discounted Markov Decision Process (MDP) [46] given by a 5-tuple (S,A, p, r, γ), where S and A are the state and action spaces, p(· | st,at) are the transition dynamics, r(st,at) is the reward function, and γ is a discount factor. ρπ(τt) denotes the state–action trajectory distribution from time t induced by a policy π(· | st). The discounted return from time step t is given by R(τt) = ∑∞ k=t γ
  kr(sk,ak) for t ∈ N0. The standard reinforcement learning objective to be maximized is the expected discounted return Jπ(τ0) = Eρπ(τ0)[R(τ0)] under the policy trajectory distribution.
sections.heading: 2.1 Improving and Accelerating Online Training via Behavioral Cloning
sections.text: We consider settings where we have a set of expert demonstrations without reward, D0 = {(sn,an)}Nn=1 = {S̄, Ā}, which we would like to use to speed up and improve online learn-
  2Code and visualizations of our results can be found at https://sites.google.com/view/nppac.
  ing [5, 42]. A standard approach for turning expert trajectories into a policy is behavioral cloning [1, 4] which involves learning a mapping from states in the expert demonstrations to their corresponding actions, that is, π0 : S → A. As such, behavioral cloning does not assume or require access to a reward function and only involves learning a mapping from states to action in a supervised fashion.
  Since expert demonstrations are costly to obtain and often only available in small number, behavioral cloning alone is typically insufficient for agents to learn good policies in complex environments and has to be complemented by a method that enables the learner to build on the cloned behavior by interacting with the environment. A particularly successful and popular class of algorithms used for incorporating behavioral policies into online training is KL-regularized reinforcement learning [10, 37, 43, 48].
sections.heading: 2.2 KL-Regularized Objectives in Reinforcement Learning
sections.text: KL-regularized reinforcement learning modifies the standard reinforcement learning objective by augmenting the return with a negative KL divergence term from the learned policy π to a reference policy π0, given a temperature parameter α. The resulting discounted return from time step t ∈ N0 is then given by
  R̃(τt) = ∞∑ k=t γk [ r(sk,ak)− αDKL(π(· | sk) ‖ π0(· | sk)) ] (1)
  and the reinforcement learning objective becomes J̃π(τ0) = Eρπ(τ0)[R̃(τ0)]. When the reference policy π0 is given by a uniform distribution, we recover the entropy-regularized reinforcement learning objective used in Soft Actor–Critic (SAC) [13] up to an additive constant.
  Under a uniform reference policy π0, the resulting objective encourages exploration, while also choosing high-reward actions. In contrast, when π0 is non-uniform, the agent is discouraged to explore areas of the state space S where the variance of π0(· | s) is low (i.e., more certain) and encouraged to explore areas of the state space where the variance of π0(· | s) is high. The KLregularized reinforcement learning objective can be optimized via policy–gradient and actor–critic algorithms.
sections.heading: 2.3 KL-Regularized Actor–Critic
sections.text: An optimal policy π that maximizes the expected KL-augmented discounted return J̃π can be learned by directly optimizing the policy gradient ∇πJ̃π. However, this policy gradient estimator exhibits high variance, which can lead to unstable learning. Actor–critic algorithms [7, 17, 32, 38] attempt to reduce this variance by making use of the state value function V π(st) = Eρπ(τt)[R̃(τt) | st] or the state–action value function Qπ(st,at) = Eρπ(τt)[R̃(τt) | st,at] to stabilize training. Given a reference policy π0(at | st), the state value function can be shown to satisfy the modified Bellman equation
  V π(st) =̇ Eat∼π(·|st)[Q π(st,at)]− αDKL ( π(· | st) ||π0(· | st) ) with a recursively defined Q-function
  Qπ(st,at) =̇ r(st,at) + γ Est+1∼p(·|st,at)[V π(st+1)].
  Instead of directly optimizing the objective function J̃π via the policy gradient, actor–critic methods alternate between policy evaluation and policy improvement [7, 13]:
  Policy Evaluation. During the policy evaluation step, Qπθ (s,a), parameterized by parameters θ, is trained by minimizing the Bellman residual
  JQ(θ) =̇ E(st,at)∼D [ (Qθ(st,at)− (r(st,at) + γEst+1∼p(·|st,at)[Vθ̄(st+1)]))2 ] , (2)
  where D is a replay buffer and θ̄ is a stabilizing moving average of parameters. Policy Improvement. In the policy improvement step, the policy πφ, parameterized by parameters φ, is updated towards the exponential of the KL-augmented Q-function,
  Jπ(φ) =̇ Est∼D [αDKL(πφ(· | st) ‖ π0(· | st))]− Est∼D [ Eat∼πφ(·|st) [Qθ(st,at)] ] , (3)
  with states sampled from a replay buffer D and actions sampled from the parameterized online policy πφ. The following sections will focus on the policy improvement objective and how certain types of references policies can lead to pathologies when optimizing Jπ(φ) with respect to φ.
sections.heading: 3 Identifying the Pathology
sections.text: In this section, we investigate the effect of KL-regularization on the training dynamics. To do so, we first consider the properties of the KL divergence to identify a potential failure mode for KL-regularized reinforcement learning. Next, we consider parametric Gaussian behavioral reference policies commonly used in practice for continuous control tasks [13, 51] and show that for Gaussian behavioral reference policies with small predictive variance, the policy improvement objective suffers from exploding gradients with respect to the policy parameters φ. We confirm that this failure occurs empirically and demonstrate that it results in slow, unstable, and suboptimal online learning. Lastly, we show that various regularization techniques used for estimating behavioral policies are unable to prevent this failure and also lead to suboptimal online policies.
sections.heading: 3.1 When Are KL-Regularized Reinforcement Learning Objectives Meaningful?
sections.text: We start by considering the properties of the KL divergence and discuss how these properties can lead to potential failure modes in KL-regularized objectives. A well-known property of KL-regularized objectives in the variational inference literature is the occurrence of singularities when the support of one distribution is not contained in the support of the other.
  To illustrate this problem, we consider the case of Gaussian behavioral and online policies commonly used in practice. Mathematically, the KL divergence between two full Gaussian distributions is always finite and well-defined. Hence, we might hope KL-regularized reinforcement learning with Gaussian behavioral and online policies to be unaffected by the failure mode described above. However, the support of a Gaussian online policy πφ(· | st) will not be contained in the support of a behavioral reference policy π0(· | st) as the predictive variance σ20(st) tends to zero, and hence DKL(πφ(· | st) ‖ π0(· | st)) → ∞ as σ20(st) → 0. In other words, as the variance of a behavioral reference policy tends to zero and the behavioral distribution becomes degenerate, the KL divergence blows up to infinity [25]. While in practice, Gaussian behavioral policy would not operate in the limit of zero variance, the functional form of the KL divergence between (univariate) Gaussians,
  DKL(πφ(· | st) ‖ π0(· | st)) ∝ log σ0(st)
  σφ(st) + σ2φ(st) + (µφ(st)− µ0(st))2 2σ20(st) ,
  implies a continuous, quadratic increase in the magnitude of the divergence as σ0(st) decreases, further exacerbated by a large difference in predictive means, |µφ(st)− µ0(st)|. As a result, for Gaussian behavioral reference policies π0(· | st) that assign very low probability to sets of points in sample space far away from the distribution’s mean µ0(st), computing the KL divergence can result in divergence values so large to cause numerical instabilities and arithmetic overflow. Hence, even for a suitably chosen behavioral reference policy class, vanishingly small behavioral reference policy predictive variances can cause the KL divergence to ‘blow up’ and cause numerical issues at evaluation points far away from states in the expert demonstrations.
  One way to address this failure mode may be to lower-bound the output of the variance network (e.g., by adding a small constant bias). However, placing a floor on the predictive variance of the behavioral reference policy is not sufficient to encourage effective learning. While it would prevent the KL divergence from blowing up, it would also lead to poor gradient signals, as well-calibrated predictive variance estimates that increase on states far away from the expert trajectories are necessary to keep the KL penalty from pulling the predictive mean of the online policy towards poor behavioral reference policy predictive means on states off the expert trajectories. Another possible solution could be to use heavy-tailed behavioral reference policies distributions, for example, Laplace distributions, to avoid pathological training dynamics. However, in Appendix B.3 we show that Laplace behavioral reference policies also suffer from pathological training dynamics, albeit less severely.
  In the following sections, we explain how an explosion in DKL(πφ(· | st) ‖ π0(· | st)) caused by small σ20(st) affects the gradients of Jπ(φ) in KL-regularized RL and discuss of how and why σ 2 0(st) may tend to zero in practice.
sections.heading: 3.2 Exploding Gradients in KL-Regularized Reinforcement Learning Objectives
sections.text: To understand how small predictive variances in behavioral reference policies can affect—and possibly destabilize—online training in KL-regularized RL, we consider the contribution of the behavioral
  reference policy’s variance to the gradient of the policy objective in Equation (3). Compared to entropy-regularized actor–critic methods (SAC, Haarnoja et al. [13]), which implicitly regularize against a uniform policy, the gradient estimator ∇̂φJπ(φ) in KL-regularized RL gains an extra scaling term ∇at log π0(at | st), the gradient of the prior log-density evaluated actions at ∼ πφ(· | s): Proposition 1 (Exploding Gradients in KL-Regularized RL). Let π0(· | s) be a Gaussian behavioral reference policy with mean µ0(st) and variance σ20(st), and let πφ(· | s) be an online policy with reparameterization at = fφ( t; st) and random vector t. The gradient of the policy loss with respect to the online policy’s parameters φ is then given by
  ∇̂φJπ(φ) = ( α∇at log πφ(at | st)− α∇at log π0(at | st) −∇atQ(st,at) ) ∇φfφ( t; st) + α∇φ log πφ(at | st)
  (4)
  with ∇at log π0(at | st) = −at−µ0(st)σ20(st) . For fixed |at − µ0(st)|, ∇at log π0(at|st) grows as O(σ−20 (st)); thus,
  | ∇̂φJπ(φ) | → ∞ as σ20(st)→ 0 whenever ∇φfφ( t; st) 6= 0.
  Proof. See Appendix A.1.
  This result formalizes the intuition presented in Section 3.1 that a behavioral reference policy with a sufficiently small predictive variance may cause KL-regularized reinforcement learning to suffer from pathological training dynamics in gradient-based optimization. The smaller the behavioral reference policy’s predictive variance, the more sensitive the policy objective’s gradients will be to differences in the means of the online and behavioral reference policies. As a result, for behavioral reference policies with small predictive variance, the KL divergence will heavily penalize online policies whose predictive means diverge from the predictive means of the behavioral policy—even in regions of the state space away from the expert trajectory where the behavioral policy’s mean prediction is poor.
sections.heading: 3.3 Predictive Uncertainty Collapse Under Parametric Policies
sections.text: The most commonly used method for estimating behavioral policies is maximum likelihood estimation (MLE) [44, 51], where we seek π0 =̇ πψ? with ψ? =̇ arg maxψ { E(s,a)∼D0 [log πψ(a | s)] } for a parametric behavioral policy πψ. In practice, πψ is often assumed to be Gaussian, πψ(· | s) = N (µψ(s),σ2ψ(s)), with µψ(s) and σ2ψ(s) parameterized by a neural network. While maximizing the likelihood of the expert trajectories under the behavioral policy is a sensible choice for behavioral cloning, the limited capacity of the neural network parameterization can produce unwanted behaviors in the resulting policy. The maximum likelihood objective ensures that the behavioral policy’s predictive mean reflects the expert’s actions and the predictive variance the (aleatoric) uncertainty inherent in the expert trajectories.
  However, the maximum likelihood objective encourages parametric policies to use their model capacity toward fitting the expert demonstrations and reflecting the aleatoric uncertainty in the data. As a result, for states off the expert trajectories, the policy can become degenerate and collapse to point predictions instead of providing meaningful predictive variance estimates that reflect that the behavioral policy ought to be highly uncertain about its predictions in previously unseen regions of the state space. Similar behaviors are well-known in parametric probabilistic models and welldocumented in the approximate Bayesian inference literature [33, 39].
  0 2 4 6 8 10 Epochs
  10−3
  10−1
  σ 2 ψ (s
  )
  Validation Variance
  Validation Log-Likelihood 0
  2 log π ψ (Ā | S̄
  )
  tently decreases during training. As shown in Proposition 1, such a collapse in predictive variance can
  result in pathological training dynamics in KL-regularized online learning—steering the online policy towards suboptimal trajectories in regions of the state space far away from the expert demonstrations and deteriorating performance.
  Effect of regularization on uncertainty collapse. To prevent a collapse in the behavioral policy’s predictive variance, prior work proposed adding entropy or Tikhonov regularization to the MLE objective [51]. However, doing so does not succeed in preventing a collapse in predictive variance off the expert demonstration trajectories, as we show in Appendix A.3. Deep ensembles [20], whose predictive mean and variance are computed from the predictive means and variances of multiple Gaussian neural networks, are a widely used method for uncertainty quantification in regression settings. However, model ensembling can be costly and unreliable, as it requires training multiple neural networks from scratch and does not guarantee well-calibrated uncertainty estimates [39, 49]. We provide visualizations in Appendix B.5 which show that ensembling multiple neural network policies does not fully prevent a collapse in predictive variance.
sections.heading: 3.4 Empirical Confirmation of Uncertainty Collapse
sections.text: To confirm Proposition 1 empirically and assess the effect of the collapse in predictive variance on the performance of KL-regularized RL, we perform an ablation study where we fix the predictive mean function of a behavioral policy to a mean function that attains 60% of the optimal performance and vary the magnitude of the policy’s predictive variance. Specifically, we set the behavioral policy’s predictive variance to different constant values in the set {1× 10−3, 5× 10−3, 1× 10−2} (following a similar implementation in Nair et al. [27]).3 The results of this experiment are shown in Figure 3, which shows the average returns, the KL divergence, and the average absolute gradients of the policy loss over training. The plots confirm that as the predictive variance of the offline behavioral policy tends to zero, the KL terms and average policy gradient magnitude explode as implied by Proposition 1, leading to unstable training and a collapse or dampening in average returns.
  In other words, even for behavioral policies with accurate predictive means, smaller predictive variances slow down or even entirely prevent learning good behavioral policies. This observation confirms that the pathology identified in Proposition 1 occurs in practice and that it can have a significant impact on KL-regularized RL from expert demonstrations, calling into question the usefulness of KL regularization as a means for accelerating and improving online training. In Appendix B.1, we show that an analogous relationship exists for the gradients of the Q-function loss.
  3We attempted to use smaller values, but the gradients grew too large and caused arithmetic overflow.
sections.heading: 4 Fixing the Pathology
sections.text: In order to address the collapse in predictive uncertainty for behavioral policies parameterized by a neural network trained via MLE, we specify a non-parametric behavioral policy whose predictive variance is guaranteed not to collapse about previously unseen states. Noting that KL-regularized RL with a behavioral policy can be viewed as approximate Bayesian inference with an empirical prior policy [13, 21, 40], we propose Non-Parametric Prior Actor–Critic (N-PPAC), an off-policy temporal difference algorithm for improved, accelerated, and stable online learning with behavioral policies.
sections.heading: 4.1 Non-Parametric Gaussian Processes Behavioral Policies
sections.text: Gaussian processes (GPs) [36] are models over functions defined by a mean m(·) and covariance function k(·, ·). When defined in terms of a non-parametric covariance function, that is, a covariance function constructed from infinitely many basis functions, we obtain a non-degenerate GP, which has sufficient capacity to prevent a collapse in predictive uncertainty away from the training data. Unlike parametric models, whose capacity is limited by their parameterization, a non-parametric model’s capacity increases with the amount of training data.
  Considering a non-parametric GP behavioral policy, π0(· | s), with A | s ∼ π0(· | s) = GP ( m(s), k(s, s′) ) , (5)
  we can obtain a non-degenerate posterior distribution over actions conditioned on the offline data D0 = {S̄, Ā} with actions sampled according to the
  A | s,D0 ∼ π0(· | s,D0) = GP ( µ0(s),Σ0(s, s ′) ) , (6)
  with
  µ(s)=m(s) + k(s, S̄)k(S̄, S̄)−1(Ā−m(Ā)) and Σ(s, s′)=k(s, s′) + k(s, S̄)k(S̄, S̄)−1k(S̄, s′). To obtain this posterior distribution, we perform exact Bayesian inference, which naively scales as O(N3) in the number of training points N , but Wang et al. [50] show that exact inference in GP regression can be scaled to N > 1, 000, 000. Since expert demonstrations usually contain less than 100k datapoints, non-parametric GP behavioral policies are applicable to a wide array of real-world tasks. For an empirical evaluation of the time complexity of using a GP prior, see Section 5.5.
  Figure 1 confirms that the non-parametric GP’s predictive variance is well-calibrated: It is small in magnitude in regions of the state space near the expert trajectories and large in magnitude in other regions of the state space. While actor–critic algorithms like SAC implicitly use a uniform prior to explore the state space, using a behavioral policy with a well-calibrated predictive variance has the benefit that in regions of the state space close to the expert demonstrations the online policy learns to match the expert, while elsewhere the predictive variance increases and encourages exploration.
  Algorithmic details. In our experiments, we use a KL-regularized objective with a standard actor– critic implementation and Double DQN [14]. Pseudocode is provided in (Appendix C.1).
sections.heading: 5 Empirical Evaluation
sections.text: We carry out a comparative empirical evaluation of our proposed approach vis-à-vis related methods that integrate offline data into online training. We provide a detailed description of the algorithms we compare against in Appendix A.4. We perform experiments on the MuJoCo benchmark suite and the substantially more challenging dexterous hand manipulation suite with sparse rewards.
  We show that KL-regularized RL with a non-parametric behavioral reference policy can rapidly learn to solve difficult high-dimensional continuous control problems given only a small set of expert demonstrations and (often significantly) outperforms state-of-the-art methods, including ones that use offline reward information—which our approach does not require. Furthermore, we demonstrate that the GP behavioral policy’s predictive variance is crucial for KL-regularized objectives to learn good online policies from expert demonstrations. Finally, we perform ablation studies that illustrate that non-parametric GP behavioral reference policies also outperform parametric behavioral reference policies with improved uncertainty quantification, such as deep ensembles and Bayesian neural
  networks (BNNs) with Monte Carlo dropout, and that the difference between non-parametric and parametric models is exacerbated the fewer expert demonstrations are available. We use the expert data from Nair et al. [27], every experiment uses six random seeds, and we use a fixed KL-temperature for each environment class. For further implementation details, see Appendix C.2.
sections.heading: 5.1 Environments
sections.text: MuJoCo locomotion tasks. We evaluate N-PPAC on three representative tasks: “Ant-v2”, “HalfCheetah-v2”, and “Walker2d-v2”. For each task, we use 15 demonstration trajectories collected by a pre-trained expert, each containing 1,000 steps. The behavioral policy is specified as the posterior distribution of a GP with a squared exponential kernel, which is well-suited for modeling smooth functions.
  Dexterous hand manipulation tasks. Real-world robot learning is a setting where human demonstration data is readily available, and many deep RL approaches fail to learn efficiently. We study this setting in a suite of challenging dexterous manipulation tasks [35] using a 28-DoF five-fingered simulated ADROIT hand. The tasks simulate challenges common to real-world settings with highdimensional action spaces, complex physics, and a large number of intermittent contact forces. We consider two tasks in particular: in-hand rotation of a pen to match a target and opening a door by unlatching and pulling a handle. We use binary rewards for task completion, which is significantly more challenging than the original setting considered in Rajeswaran et al. [35]. 25 expert demonstrations were provided for each task, each consisting of 200 environment steps which are not fully optimal but do successfully solve the task. The behavioral policy is specified as the posterior distribution of a GP with a Matérn kernel, which is more suitable for modeling non-smooth data.
sections.heading: 5.2 Results
sections.text: On MuJoCo environments, KL-regularized RL with a non-parametric behavioral policy consistently outperforms all related methods across all three tasks, successfully accelerating learning from offline data, as shown in Figure 4. Most notably, it outperforms methods such as AWAC [27]—the previous state-of-the-art—which attempts to eschew the problem of learning behavioral policies but instead uses an implicit constraint. Our approach, N-PPAC, exhibits an increase in stability and higher returns compared to comparable methods such as ABM and BRAC that explicitly regularize the online policy against a parametric behavioral policy and plateau at suboptimal performance levels as they are being forced to copy poor actions from the behavioral policy away from the expert data. In contrast, using a non-parametric behavioral policy allows us to avoid such undesirable behavior.
  On dexterous hand manipulation environments, KL-regularized RL with a non-parametric behavioral policy performs on par or outperforms all related methods on both tasks, as shown in Figure 5. Most notably, on the door opening task, it achieves a stable success rate of 90% within only 100,000 environment interactions For comparison, AWAC requires 4× as many environment interactions to
  achieve the same performance and is significantly less stable, while most other methods fail to learn any meaningful behaviors.
  Alternative divergence metrics underperform KL-regularization. KL-regularized RL with a non-parametric behavioral policy consistently outperforms methods that use alternative divergence metrics, as shown in the bottom plots of Figures 4 and 5.
sections.heading: 5.3 Can the Pathology Be Fixed by Improved Parametric Uncertainty Quantification?
sections.text: line policy success rates. We consider the challenging “door-binary-v0” environment for this ablation study.
  Parametric uncertainty quantification is insufficient. Figure 5 shows that parametric variance functions result in online policies that only achieve success rates of up to 20% and eventually deteriorate, whereas the non-parametric variance yields an online policy that achieves a success rate of nearly 100%. This finding shows that commonly used uncertainty quantification methods, such as deep ensembles or BNNs with Monte Carlo dropout, do not generate sufficiently well-calibrated uncertainty estimates to remedy the pathology, and better methods may be needed [9, 39, 41].
  Lower-bounding the predictive variance does not remedy the pathology. The predictive variance of all MLE-based and ensemble behavioral reference policies in all experiments are bounded away from zero at a minimum value of ≈ 10−2. Hence, setting a floor on the variance is not sufficient to prevent pathological training dynamics. This result further demonstrates the importance of accurate predictive variance estimation in allowing the online policy to match expert actions in regions of the state space with low behavioral policy predictive variance and explore elsewhere.
sections.heading: 5.4 Can a Single Expert Demonstration Be Sufficient to Accelerate Online Training?
sections.text: To assess the usefulness of nonparametric behavioral reference policies in settings where only few expert demonstrations are available, we investigate whether the difference in performance between online policies trained with non-parametric and parametric behavioral reference policies, respectively, is exacerbated the fewer expert demonstrations are available. To answer this question, we consider the “HalfCheetah-v2” environment and compare online policies trained with different behavioral reference
  policies—non-parametric GPs, deep ensembles, and BNNs with Monte Carlo dropout—estimated either from 15 expert demonstrations (i.e., 15 state–action trajectories, containing 15,000 samples) or from a single expert demonstration (i.e., a single state–action trajectory, containing 1,000 samples).
sections.heading: A single expert demonstration is sufficient for non-parametric behavioral reference policies.
sections.text: Figure 7 shows the returns for online policies trained with behavioral reference policies estimated from the full dataset (top plot) and from only a single expert state–action trajectory (bottom plot). On the full dataset, we find that all three methods are competitive and improve on the prior stateof-the-art but that the GP behavioral policy leads to the highest return. Remarkably, non-parametric GP behavioral policies perform just as well with only a single expert demonstration as with all 15 (i.e., with 1,000 data points, instead of 15,000 data points). These results further emphasizes the usefulness of non-parametric behavioral policies when accelerating online training with expert demonstrations—even when only very few expert demonstrations are available.
sections.heading: 5.5 Are Non-Parametric GP Behavioral Reference Policies Too Computationally Expensive?
sections.text: Table 1 presents the time complexity of KL-regularized RL under non-parametric GP and parametric neural network behavioral reference policies, as measured by the average time elapsed per epoch on the “door-binary-v0” and “HalfCheetah-v2” environments. One epoch of online training on “doorbinary-v0” and “HalfCheetah-v2” requires computing the KL divergence over 1,000 mini-batches of size 256 and 1,024, respectively. The time complexity of evaluating the log-density of a GP behavioral reference policy—needed for computing gradients of the KL divergence during online training—scales quadratically in the number of training data points and linearly in the dimensionality of the state and action space, respectively. As can be seen in Table 1, non-parametric GP behavioral reference policies only lead to a modest increase in the time needed to complete one epoch of training while resulting in significantly improved performance as shown in Figures 4 and 5.
sections.heading: 6 Conclusion
sections.text: We identified a previously unrecognized pathology in KL-regularized RL from expert demonstrations and showed that this pathology can significantly impede and even entirely prevent online learning. To remedy the pathology, we proposed the use of non-parametric behavioral reference policies, which we showed can significantly accelerate and improve online learning and yield online policies that (often significantly) outperform current state-of-the-art methods on challenging continuous control tasks. We hope that this work will encourage further research into better model classes for deep reinforcement learning algorithms, including and especially for reinforcement from image inputs.
  Acknowledgments and Disclosure of Funding
  We thank Ashvin Nair for sharing his code and results, as well as for providing helpful insights about the dexterous hand manipulation suite. We also thank Clare Lyle, Charline Le Lan, and Angelos Filos for detailed feedback on an early draft of this paper, Avi Singh for early discussions about behavioral cloning in entropy-regularized RL, and Tim Pearce for a useful discussion on the role of good models in RL. TGJR and CL are funded by the Engineering and Physical Sciences Research Council (EPSRC). TGJR is also funded by the Rhodes Trust and by a Qualcomm Innovation Fellowship. We gratefully acknowledge donations of computing resources by the Alan Turing Institute.
title: On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations