summaries: This paper presents a deep reinforcement algorithm, Ensemble Deep Deterministic Policy Gradients (ED2), for continuous control tasks. The algorithm is empirically derived and is claimed to represent SotA performance on several tasks and while providing more stable results. These claims are justified based primarily on the (reward and stability) results on 4 MuJoCo environments.   This paper has two main contributions: it introduces an ensemble-based actor-critic method, and it answers some pertinent questions in policy optimization by focusing on its different components. The ensemble is different from multi-actor learners that interact with multiple environments simultaneously, violating the standard RL setup. Instead, the learner of this paper maintains multiple actors and critics but uses only a single actor at a time to interact with the environment. All actors and critics are trained on a common replay buffer. The base method is the streamlined off-policy (SOP) method, which unlike soft actor-critic (SAC) doesn’t use an entropy bonus. Additionally, no exploration noise is added, resulting in their Ensemble Deep Deterministic (ED2) method.  The proposed algorithm ED2 is shown to be superior and more stable in performance according to different measures compared to existing methods. It is also revealed that actor initialization affects performance less than critic initialization. ED2 uses deterministic actors, and its exploration comes from sampling among the actors. Such a form of exploration is also shown to be superior to UCB-style exploration.  The paper presents an empirical study evaluating the commonly accepted design choice in off-policy Deep RL algorithms in continuous control settings. The use of additive exploration noise, initialization choices, update frequency, and precision for retraining are tested empirically highlighting some interesting results. The paper also introduces ED2 - an ensemble method utilizing the design choices from the study which is demonstrated to achieve SOTA results on Mujoco benchmarks. This paper conducted an experimental study over a range of tricks that are often exploited to facilitate ensemble deep reinforcement learning. The experiment results show several interesting findings. For example, it was found that commonly used additive action noise may not be necessary for effective exploration. Meanwhile, experiments show that the initialization of critics perhaps has a higher impact on learning performance than the initialization methods adopted for actors. These findings can be quite important to guide future design of more effective ensemble reinforce learning algorithms.
abstractText: The growth of deep reinforcement learning (RL) has brought multiple exciting tools and methods to the field. This rapid expansion makes it important to understand the interplay between individual elements of the RL toolbox. We approach this task from an empirical perspective by conducting a study in the continuous control setting. We present multiple insights of fundamental nature, including: a commonly used additive action noise is not required for effective exploration and can even hinder training; the performance of policies trained using existing methods varies significantly across training runs, epochs of training, and evaluation runs; the critics’ initialization plays the major role in ensemble-based actor-critic exploration, while the training is mostly invariant to the actors’ initialization; a strategy based on posterior sampling explores better than the approximated UCB combined with the weighted Bellman backup; the weighted Bellman backup alone cannot replace the clipped double Q-Learning. As a conclusion, we show how existing tools can be brought together in a novel way, giving rise to the Ensemble Deep Deterministic Policy Gradients (ED2) method, to yield state-of-the-art results on continuous control tasks from OpenAI Gym MuJoCo. From the practical side, ED2 is conceptually straightforward, easy to code, and does not require knowledge outside of the existing RL toolbox.
references.authors: Joshua Achiam
references.title: Spinning Up in Deep Reinforcement Learning
references.venue: GitHub repository,
references.authors: Marcin Andrychowicz Anton Raichuk Piotr Stanczyk Manu Orsini Sertan Girgin Raphael Marinier Léonard Hussenot Matthieu Geist Olivier Pietquin Marcin Michalski Sylvain Gelly Olivier Bachem
references.title: What Matters in On-Policy Reinforcement Learning
references.venue: A Large-Scale Empirical Study,
references.authors: OpenAI: Marcin Andrychowicz Bowen Baker Maciek Chociej Rafal Jozefowicz Bob McGrew Jakub Pachocki Arthur Petron Matthias Plappert Glenn Powell Alex Ray
references.title: Learning dexterous in-hand manipulation
references.venue: The International Journal of Robotics Research,
references.authors: Marc G. Bellemare Sriram Srinivasan Georg Ostrovski Tom Schaul David Saxton Rémi Munos
references.title: Unifying count-based exploration and intrinsic motivation
references.venue: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016,
references.authors: Christopher Berner Greg Brockman Brooke Chan Vicki Cheung Przemysław Dębiak Christy Dennison David Farhi Quirin Fischer Shariq Hashme Chris Hesse
references.title: Dota 2 with large scale deep reinforcement learning
references.venue: arXiv preprint arXiv:1912.06680,
references.authors: Stephanie C.Y. Chan Samuel Fishman Anoop Korattikara John Canny Sergio Guadarrama
references.title: Measuring the Reliability of Reinforcement Learning Algorithms
references.venue: In 8th International Conference on Learning Representations,
references.authors: Xi Liang Chen Lei Cao Chen Xi Li Zhi Xiong Xu Jun Lai
references.title: Ensemble Network Architecture for Deep Reinforcement Learning
references.venue: Mathematical Problems in Engineering,
references.authors: Kurtland Chua Roberto Calandra Rowan McAllister Sergey Levine
references.title: Deep reinforcement learning in a handful of trials using probabilistic dynamics models
references.venue: NeurIPS
references.authors: Stanislav Fort Huiyi Hu Balaji Lakshminarayanan
references.title: Deep ensembles: A loss landscape perspective
references.venue: CoRR, abs/1912.02757,
references.authors: Meire Fortunato Mohammad Gheshlaghi Azar Bilal Piot Jacob Menick Matteo Hessel Ian Osband Alex Graves Vlad Mnih Remi Munos Demis Hassabis Olivier Pietquin Charles Blundell Shane Legg
references.title: Noisy networks for exploration
references.venue: In 6th International Conference on Learning Representations,
references.authors: Scott Fujimoto Herke van Hoof David Meger
references.title: Addressing function approximation error in actor-critic methods
references.venue: Proceedings of the 35th International Conference on Machine Learning,
references.authors: Tuomas Haarnoja Aurick Zhou Pieter Abbeel Sergey Levine
references.title: Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor
references.venue: In 35th International Conference on Machine Learning,
references.authors: Tuomas Haarnoja Aurick Zhou Kristian Hartikainen George Tucker Sehoon Ha Jie Tan Vikash Kumar Henry Zhu Abhishek Gupta Pieter Abbeel Sergey Levine
references.title: Soft actor-critic algorithms and applications
references.venue: CoRR, abs/1812.05905,
references.authors: Matteo Hessel Joseph Modayil Hado Van Hasselt Tom Schaul Georg Ostrovski Will Dabney Dan Horgan Bilal Piot Mohammad Azar David Silver
references.title: Rainbow: Combining improvements in deep reinforcement learning
references.venue: AAAI Conference on Artificial Intelligence,
references.authors: Zhewei Huang Shuchang Zhou Bo Er Zhuang Xinyu Zhou
references.title: Learning to run with actor-critic ensemble, 2017
references.authors: Andrew Ilyas Logan Engstrom Shibani Santurkar Dimitris Tsipras Firdaus Janoos Larry Rudolph Aleksander Madry
references.title: A closer look at deep policy gradients
references.venue: In 8th International Conference on Learning Representations,
references.authors: Diederik P. Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations,
references.authors: Aviral Kumar Abhishek Gupta Sergey Levine
references.title: Discor: Corrective feedback in reinforcement learning via distribution correction
references.venue: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems
references.authors: Thanard Kurutach Ignasi Clavera Yan Duan Aviv Tamar Pieter Abbeel
references.title: Model-ensemble trust-region policy optimization
references.venue: In 6th International Conference on Learning Representations,
references.authors: Tze Leung Lai Herbert Robbins
references.title: Asymptotically efficient adaptive allocation rules
references.venue: Advances in applied mathematics,
references.authors: Balaji Lakshminarayanan Alexander Pritzel Charles Blundell
references.title: Simple and scalable predictive uncertainty estimation using deep ensembles
references.venue: In NIPS 2017,
references.authors: Kimin Lee Michael Laskin Aravind Srinivas Pieter Abbeel
references.title: SUNRISE: A Simple Unified Framework for Ensemble
references.venue: Learning in Deep Reinforcement Learning,
references.authors: Timothy P. Lillicrap Jonathan J. Hunt Alexander Pritzel Nicolas Heess Tom Erez Yuval Tassa David Silver Daan Wierstra
references.title: Continuous control with deep reinforcement learning
references.venue: In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations,
references.authors: Kendall Lowrey Aravind Rajeswaran Sham Kakade Emanuel Todorov Igor Mordatch
references.title: Plan online, learn offline: Efficient learning and exploration via model-based control
references.authors: Piotr Miłoś Łukasz Kuciński Konrad Czechowski Piotr Kozakowski Maciek Klimek
references.title: Uncertainty-sensitive learning and planning with ensembles
references.venue: arXiv preprint arXiv:1912.09996,
references.authors: Ian Osband Benjamin Van Roy
references.title: Why is posterior sampling better than optimism for reinforcement learning
references.venue: In 34th International Conference on Machine Learning,
references.authors: Ian Osband Benjamin Van Roy Daniel Russo
references.title: More) efficient reinforcement learning via posterior sampling
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ian Osband Charles Blundell Alexander Pritzel Benjamin Van Roy
references.title: Deep exploration via bootstrapped DQN
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ian Osband John Aslanides Albin Cassirer
references.title: Randomized prior functions for deep reinforcement learning
references.venue: In NeurIPS,
references.authors: Ian Osband Benjamin Van Roy Daniel J. Russo Zheng Wen
references.title: Deep exploration via randomized value functions
references.venue: J. Mach. Learn. Res.,
references.authors: Matthias Plappert Rein Houthooft Prafulla Dhariwal Szymon Sidor Richard Y. Chen Xi Chen Tamim Asfour Pieter Abbeel Marcin Andrychowicz
references.title: Parameter space noise for exploration
references.venue: In 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings,
references.authors: David Silver Aja Huang Chris J Maddison Arthur Guez Laurent Sifre George Van Den Driessche Julian Schrittwieser Ioannis Antonoglou Veda Panneershelvam Marc Lanctot
references.title: Mastering the game of go with deep neural networks and tree
references.venue: search. nature,
references.authors: Malcolm J.A. Strens
references.title: A bayesian framework for reinforcement learning
references.venue: In Proceedings of the Seventeenth International Conference on Machine Learning (ICML
references.authors: Hado van Hasselt Arthur Guez David Silver
references.title: Deep reinforcement learning with double qlearning
references.venue: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17,
references.authors: Oriol Vinyals Igor Babuschkin Wojciech M Czarnecki Michaël Mathieu Andrew Dudzik Junyoung Chung David H Choi Richard Powell Timo Ewalds Petko Georgiev
references.title: Grandmaster level in starcraft ii using multi-agent reinforcement learning
references.authors: Che Wang Yanqiu Wu Quan Vuong Keith Ross
references.title: Striving for simplicity and performance in off-policy DRL: Output normalization and non-uniform sampling
references.venue: Proceedings of the 37th International Conference on Machine Learning,
references.authors: Marco A. Wiering Hado van Hasselt
references.title: Ensemble algorithms in reinforcement learning
references.venue: IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics,
references.authors: Zeyu Zheng Junhyuk Oh Satinder Singh
references.title: On Learning Intrinsic Rewards for Policy Gradient Methods
references.venue: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems
references.authors: Zhuobin Zheng Chun Yuan Zhihui Lin Yangyang Cheng Hanghao Wu
references.title: Self-adaptive double bootstrapped DDPG
references.venue: In IJCAI International Joint Conference on Artificial Intelligence, 2018b. ISBN 9780999241127
references.authors: Wang
references.title: The average test returns across the training of our method and ED2 with the UCB exploration. Gaussian noise While our method uses ensemble-based temporally coherent exploration, the most popular choice of exploration is injecting i.i.d. noise (Fujimoto et al., 2018
sections.heading: 1 INTRODUCTION
sections.text: Recently, deep reinforcement learning (RL) has achieved multiple breakthroughs in a range of challenging domains (e.g. Silver et al. (2016); Berner et al. (2019); Andrychowicz et al. (2020b); Vinyals et al. (2019)). A part of this success is related to an ever-growing toolbox of tricks and methods that were observed to boost the RL algorithms’ performance (e.g. Hessel et al. (2018); Haarnoja et al. (2018b); Fujimoto et al. (2018); Wang et al. (2020); Osband et al. (2019)). This state of affairs benefits the field but also brings challenges related to often unclear interactions between the individual improvements and the credit assignment related to the overall performance of the algorithm Andrychowicz et al. (2020a); Ilyas et al. (2020).
  In this paper, we present a comprehensive empirical study of multiple tools from the RL toolbox applied to the continuous control in the OpenAI Gym MuJoCo setting. These are presented in Section 4 and Appendix B. Our insights include:
  • The normally distributed action noise, commonly used for exploration, hinders training. • The current state-of-the-art methods are unstable under several stability criteria. • The critics’ initialization plays a major role in ensemble-based actor-critic exploration, while
  the training is mostly invariant to the actors’ initialization. • The approximated posterior sampling exploration (Osband et al., 2013) outperforms approx-
  imated UCB exploration combined with weighted Bellman backup (Lee et al., 2020). • The weighted Bellman backup (Lee et al., 2020) can not replace the clipped double Q-
  Learning (Fujimoto et al., 2018). To address some of the issues listed above, we introduce the Ensemble Deep Deterministic Policy Gradient (ED2) algorithm1, see Section 3. ED2 brings together existing RL tools in a novel way: it is
  1Our code is based on SpinningUp (Achiam, 2018). We open-source it at: https://github.com/ ed2-paper/ED2.
  an off-policy algorithm for continuous control, which constructs an ensemble of streamlined versions of TD3 agents and achieves the state-of-the-art performance in OpenAI Gym MuJoCo, substantially improving the results on the two hardest tasks – Ant and Humanoid. Consequently, ED2 does not require knowledge outside of the existing RL toolbox, is conceptually straightforward, and easy to code.
sections.heading: 2 BACKGROUND
sections.text: We model the environment as a Markov Decision Process (MDP). It is defined by the tuple (S,A, R, P, γ, p0), where S is a continuous multi-dimensional state space, A denotes a continuous multi-dimensional action space, P is a transition kernel, γ ∈ [0, 1) stands for a discount factor, p0 refers to an initial state distribution, and R is a reward function. The agent learns a policy from sequences of transitions τ = [(st, at, rt, st+1, d)]Tt=0, called episodes or trajectories, where at ∼ π(·|st), st+1 ∼ P (·|st, at), rt = R(st, at, st+1), d is a terminal signal, and T is the terminal time-step. A stochastic policy π(a|s) maps each state to a distribution over actions. A deterministic policy µ : S −→ A assigns each state an action. All algorithms that we consider in this paper use a different policy for collecting data (exploration) and a different policy for evaluation (exploitation). In order to keep track of the progress, the evaluation runs are performed every ten thousand environment interactions. Because of the environments’ stochasticity, we run the evaluation policy multiple times. Let {Ri}Ni=1 be a set of (undiscounted) returns from N evaluation episodes {τi}Ni=1, i.e. Ri = ∑ rt∈τi rt. We evaluate the
  policy using the average test return R̄ = 1N ∑N i=1Ri and the standard deviation of the test returns
  σ = √
  1 N−1 ∑N i=1(Ri − R̄)2.
  We run experiments on four continuous control tasks and their variants, introduced in the appropriate sections, from the OpenAI Gym MuJoCo suite (Brockman et al., 2016) presented in Figure 1. The agent observes vectors that describe the kinematic properties of the robot and its actions specify torques to be applied on the robot joints. See Appendix D for the details on the experimental setup.
sections.heading: 3 ENSEMBLE DEEP DETERMINISTIC POLICY GRADIENTS
sections.text: For completeness of exposition, we present ED2 before the experimental section. The ED2 architecture is based on an ensemble of Streamlined Off-Policy (SOP) agents (Wang et al., 2020), meaning that our agent is an ensemble of TD3-like agents (Fujimoto et al., 2018) with the action normalization and the ERE replay buffer. The pseudo-code listing can be found in Algorithm 1, while the implementation details, including a more verbose version of pseudo-code (Algorithm 3), can be found in Appendix E. In the data collection phase (Lines 1-9), ED2 selects one actor from the ensemble uniformly at random (Lines 1 and 9) and run its deterministic policy for the course of one episode (Line 4). In the evaluation phase (not shown in Algorithm 1), the evaluation policy averages all the actors’ output actions. We train the ensemble every 50 environment steps with 50 stochastic gradient descent updates (Lines 10-13). ED2 concurrently learns K · 2 Q-functions, Qφk,1 and Qφk,2 where k ∈ K, by mean square Bellman error minimization, in almost the same way that SOP learns its two Q-functions. The only difference is that we have K critic pairs that are initialized with different random weights and then trained independently with the same batches of data. Because of the different initial weights, each Q-function has a different bias in its Q-values. The K actors, πθk , train maximizing their corresponding first critic, Qφk,1 , just like SOP.
  Algorithm 1 ED2 - Ensemble Deep Deterministic Policy Gradients Input: init. params for policy θk and Q-functions φk,1, φk,2, k ∈ [1...K]; replay buffer D;
  1: Sample the current policy index c ∼ U([1...K]). 2: Reset the environment and observe the state s. 3: repeat 4: Execute action a = µθc(s) . µ uses the action normalization 5: Observe and store (s, a, r, s′, d) in the replay buffer D. 6: Set s← s′ 7: if episode is finished then 8: Reset the environment and observe initial state s. 9: Sample the current policy index c ∼ U([1...K]).
  10: if time to update then 11: for as many as steps done in the environment do 12: Sample a batch of transitions B = {(s, a, r, s′, d)} ⊂ D . uses ERE 13: Update the parameters θk, φk,1 and φk,2 by one gradient step. 14: until convergence
  Utilizing the ensembles requires several design choices, which we summarize below. The ablation study of ED2 elements is provided in Appendix C.
sections.heading: Ensemble
sections.text: Used: We train the ensemble of 5 actors and 5 critics; each actor learns from its own critic and the whole ensemble is trained on the same data.
  Not used: We considered different actor-critic configurations, initialization schemes and relations, as well as the use of random prior networks (Osband et al., 2018), data bootstrap (Osband et al., 2016), and different ensemble sizes. We also change the SOP network sizes and training intensity instead of using the ensemble. Besides the prior networks in some special cases, these turn out to be inferior as shown in Section 4 and Appendix B.1.
sections.heading: Exploration
sections.text: Used: We pick one actor uniformly at random to collect the data for the course of one episode. The actor is deterministic (no additive action noise is applied). These two choices ensure coherent and temporally-extended exploration similarly to Osband et al. (2016).
  Not used: We tested several approaches to exploration: using the ensemble of actors, UCB (Lee et al., 2020), and adding the action noise in different proportions. These experiments are presented in Appendix B.2.
sections.heading: Exploitation
sections.text: Used: The evaluation policy averages all the actors’ output actions to provide stable performance.
  Not used: We tried picking an action with the biggest value estimate (average of the critics’ Qfunctions) in evaluation (Huang et al., 2017).
  Interestingly, both policies had similar results, see Appendix B.3.
sections.heading: Action normalization
sections.text: Used: We use the action normalization introduced by Wang et al. (2020).
  Not used: We experimented with the observations and rewards normalization, which turned out to be unnecessary. The experiments are presented in Appendix B.4.
sections.heading: Q-function updates
sections.text: Used: We do 50 SGD updates (ADAM optimizer (Kingma and Ba, 2015), MSE loss) to the actors and the critics every 50 environment interactions, use Clipped Double Q-Learning (Fujimoto et al., 2018).
  Not used: We also examined doing the updates at the end of each episode (with the proportional number of updates), using the Hubert loss, and doing weighted Bellman backups (Lee et al., 2020). However, we found them to bring no improvement to our method, as presented in Appendix B.5.
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we present our comprehensive study and the resulting insights. The rest of the experiments verifying that our design choices perform better than alternatives are in Appendix B. Unless stated otherwise, a solid line in the figures represents an average, while a shaded region shows a 95% bootstrap confidence interval. We used 30 seeds for ED2 and the baselines and 7 seeds for the ED2 variants.
sections.heading: 4.1 THE NORMALLY DISTRIBUTED ACTION NOISE, COMMONLY USED FOR EXPLORATION, HINDERS TRAINING
sections.text: In this experiment, we deprive SOP of its exploration mechanism, namely additive normal action noise, and call this variant deterministic SOP (det. SOP). It causes relatively minor deterioration in the Humanoid performance, has no significant influence on the Hopper or Walker performance, and substantially improves the Ant performance, see Figure 2. This result shows that no additional exploration mechanism, often in a form of an exploration noise (Lillicrap et al., 2016; Fujimoto et al., 2018; Wang et al., 2020), is required for the diverse data collection and it can even hinder training.
  ED2 leverages this insight and constructs an ensemble of deterministic SOP agents presented in Section 3. Figure 3 shows that ED2 magnifies the beneficial effect coming from the deterministic exploration.
  ED2 achieves state-of-the-art performance on the OpenAI Gym MuJoCo suite. Figure 4 shows the results of ED2 contrasted with three strong baselines: SUNRISE (Lee et al., 2020), SOP (Wang et al., 2020), and SAC(Haarnoja et al., 2018b).
  For completeness, we plot the Humanoid velocities in Figure 5 which shows that our method accelerates to a much higher velocity than the baselines.
sections.heading: 4.2 THE CURRENT STATE-OF-THE-ART METHODS ARE UNSTABLE UNDER SEVERAL STABILITY CRITERIA
sections.text: We consider three notions of stability: inference stability, asymptotic performance stability, and training stability. ED2 outperforms baselines in each of these notions, as discussed below. Similar metrics were also studied in Chan et al. (2020).
  Inference stability We say that an agent is inference stable if, when run multiple times, it achieves similar test performance every time. We measure inference stability using the standard deviation of test returns explained in Section 2. We found that that the existing methods train policies that are surprisingly sensitive to the randomness in the environment initial conditions2. Figure 4 and Figure 6 show that ED2 successfully mitigates this problem. By the end of the training, ED2 produces results within 1% of the average performance on Humanoid, while the performance of SUNRISE, SOP, and SAC may vary as much as 11%.
  2The MuJoCo suite is overall deterministic, nevertheless, little stochasticity is injected at the beginning of each trajectory, see Appendix D for details.
  Asymptotic performance stability We say that an agent achieves asymptotic performance stability if it achieves similar test performance across multiple training runs starting from different initial networks weights. Figure 7 shows that ED2 has a significantly smaller variance than the other methods while maintaining high performance.
  Training stability We will consider training stable if performance does not severely deteriorate from one evaluation to the next. We define the root mean squared deterioration metric (RMSD) as follows:
  RMSD = √√√√ 1 M M∑ i=1 ( max(R̄i−20 − R̄i, 0) )2 ,
  where M is the number of the evaluation phases during training and R̄i is the average test return at the i-th evaluation phase (described in Section 2). We compare returns 20 evaluation phases apart to ensure that the deterioration in performance doesn’t stem from the evaluation variance. ED2 has the lowest RMSD across all tasks, see Figure 8.
sections.heading: 4.3 THE CRITICS’ INITIALIZATION PLAYS A MAJOR ROLE IN ENSEMBLE-BASED ACTOR-CRITIC EXPLORATION, WHILE THE TRAINING IS MOSTLY INVARIANT TO THE
sections.text: ACTORS’ INITIALIZATION
  In this experiment, actors’ weights are initialized with the same random values (contrary to the standard case of different initialization). Moreover, we test a corresponding case with critics’ weights initialized with the same random values or simply training only a single critic.
  Figure 9 indicates that the choice of actors initialization does not matter in all tasks but Humanoid. Although the average performance on Humanoid seems to be better, it is also less stable. This is quite interesting because the actors are deterministic. Therefore, the exploration must come from the fact that each actor is trained to optimize his own critic.
  On the other, Figure 9 shows that the setup with the single critic severely impedes the agent performance. We suspect that using the single critic impairs the agent exploration capabilities as its actors’ policies, trained to maximize the same critic’s Q-function, become very similar.
sections.heading: 4.4 THE APPROXIMATED POSTERIOR SAMPLING EXPLORATION OUTPERFORMS APPROXIMATED UCB EXPLORATION COMBINED WITH WEIGHTED BELLMAN BACKUP
sections.text: ED2 uses posterior sampling based exploration method (Osband et al., 2016). SUNRISE, on the other hand, approximates the Upper Confidence Bound (UCB) exploration technique and does weighted Bellman backups (Lee et al., 2020). For the fair comparison between ED2 and SUNRISE, we substitute the SUNRISE base algorithm SAC for the SOP algorithm used by ED2. We call this variant SUNRISE-SOP.
  We test both methods on the standard MuJoCo benchmarks as well as delayed (Zheng et al., 2018a) and sparse (Plappert et al., 2018) rewards variants. Both variations make the environments harder from the exploration standpoint. In the delayed version, the rewards are accumulated and returned to the agent only every 10 time-steps. In the sparse version, the reward for the forward motion is returned to the agent only after it crosses the threshold of one unit on the x-axis. For a better perspective, a fully trained Humanoid is able to move to around five units until the end of the episode. All the other reward components (living reward, control cost, and contact cost) remain unchanged. The results are presented in Table 1.
  ED2 outperforms the non-ensemble method SOP, supporting the argument of coherent and temporallyextended exploration of ED2. Moreover, we observe that performance in MuJoCo environments benefits from the ED2 approximate Bayesian posterior sampling exploration (Osband et al., 2013) in contrast to the approximated UCB in SUNRISE, which follows the OFU principle. The posterior sampling is proved to be theoretically superior to the OFU strategy (Osband and Van Roy, 2017).
  The experiment where the ED2’s exploration mechanism is replaced for UCB is in Appendix B.2. This variant also achieves worse results than ED2. The additional exploration efficiency experiment in the custom Humanoid environment, where an agent has to find and reach a goal position, is in Appendix A.
sections.heading: 4.5 THE WEIGHTED BELLMAN BACKUP CAN NOT REPLACE THE CLIPPED DOUBLE Q-LEARNING
sections.text: We applied the weighted Bellman backups proposed by Lee et al. (2020) to our method. It is suggested that the method mitigates error propagation in Q-learning by re-weighting the Bellman backup based on uncertainty estimates from an ensemble of target Q-functions (i.e. variance of predictions). Interestingly, Figure 10 does not show this positive effect on ED2.
  Our method uses clipped double Q-Learning to mitigate overestimation in Q-functions (Fujimoto et al., 2018). We wanted to check if it is required and if it can be exchanged for the weighted Bellman backups used by Lee et al. (2020). Figure 11 shows that clipped double Q-Learning is required and that the weighted Bellman backups can not replace it.
sections.heading: 5 RELATED WORK
sections.text: Off-policy RL Recently, multiple deep RL algorithms for continuous control have been proposed, e.g. DDPG (Lillicrap et al., 2016), TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018b), SOP (Wang et al., 2020), SUNRISE (Lee et al., 2020). They provide a variety of methods for improving training quality, including double-Q bias reduction van Hasselt et al. (2016), target policy smoothing or different update frequencies for actor and critic Fujimoto et al. (2018), entropy regularization Haarnoja et al. (2018b), action normalization Wang et al. (2020), prioritized experience replay Wang et al. (2020), weighted Bellman backups Kumar et al. (2020); Lee et al. (2020), and use of ensembles Osband et al. (2019); Lee et al. (2020); Kurutach et al. (2018); Chua et al. (2018).
  Ensembles Deep ensembles are a practical approximation of a Bayesian posterior, offering improved accuracy and uncertainty estimation Lakshminarayanan et al. (2017); Fort et al. (2019). They
  inspired a variety of methods in deep RL. They are often used for temporally-extended exploration; see the next paragraph. Other than that, ensembles of different TD-learning algorithms were used to calculate better Q-learning targets (Chen et al., 2018). Others proposed to combine the actions and value functions of different RL algorithms Wiering and van Hasselt (2008) or the same algorithm with different hyper-parameters Huang et al. (2017). For mixing the ensemble components, complex self-adaptive confidence mechanisms were proposed in Zheng et al. (2018b). Our method is simpler: it uses the same algorithm with the same hyper-parameters without any complex or learnt mixing mechanism. Lee et al. (2020) proposed a unified framework for ensemble learning in deep RL (SUNRISE) which uses bootstrap with random initialization Osband et al. (2016) similarly to our work. We achieve better results than SUNRISE and show in Appendix B that their UCB exploration and weighted Bellman backups do not aid our algorithm performance.
  Exploration Various frameworks have been developed to balance exploration and exploitation in RL. The optimism in the face of uncertainty principle Lai and Robbins (1985); Bellemare et al. (2016) assigns an overly optimistic value to each state-action pair, usually in the form of an exploration bonus reward, to promote visiting unseen areas of the environment. The maximum entropy method Haarnoja et al. (2018a) encourages the policy to be stochastic, hence boosting exploration. In the parameter space approach Plappert et al. (2018); Fortunato et al. (2018), noise is added to the network weights, which can lead to temporally-extended exploration and a richer set of behaviours. Posterior sampling Strens (2000); Osband et al. (2016; 2018) methods have similar motivations. They stem from the Bayesian perspective and rely on selecting the maximizing action among sampled and statistically plausible set of action values. The ensemble approach Lowrey et al. (2018); Miłoś et al. (2019); Lee et al. (2020) trains multiple versions of the agent, which yields a diverse set of behaviours and can be viewed as an instance of posterior sampling RL.
sections.heading: 6 CONCLUSIONS
sections.text: We conduct a comprehensive empirical analysis of multiple tools from the RL toolbox applied to the continuous control in the OpenAI Gym MuJoCo setting. We believe that the findings can be useful to RL researchers. Additionally, we propose Ensemble Deep Deterministic Policy Gradients (ED2), an ensemble-based off-policy RL algorithm, which achieves state-of-the-art performance and addresses several issues found during the aforementioned study.
sections.heading: 7 REPRODUCIBILITY STATEMENT
sections.text: We have made a significant effort to make our results reproducible. We use 30 random seeds, which is above the currently popular choice in the field (up to 5 seeds). Furthermore, we systematically explain our design choices in Section 3 and we provide a detailed pseudo-code of our method in Algorithm 3 in the Appendix B. Additionally, we open-sourced the code for the project3 together with examples of how to reproduce the main experiments. The implementation details are explained in Appendix E and extensive information about the experimental setup is given in Appendix D.
  3https://github.com/ed2-paper/ED2
sections.heading: A EXPLORATION EFFICIENCY IN THE CUSTOM HUMANOID ENVIRONMENT
sections.text: To check the exploration capabilities of our method, we constructed two environments based on Humanoid where the goal is not only to move forward as fast as possible but to find and get to the specific region. The environments are described in Figure 12.
  Because the Humanoid initial state is slightly perturbed every run, we compare solved rates over multiple runs, see details in Appendix D. Figure 13 compares the solved rates of our method and the three baselines. Our method outperforms the baselines. For this experiment, our method uses the prior networks (Osband et al., 2018).
sections.heading: B DESIGN CHOICES
sections.text: In this section, we summarize the empirical evaluation of various design choices grouped by topics related to an ensemble of agents (B.1), exploration (B.2), exploitation (B.3), normalization (B.4), and Q-function updates (B.5). In the plots, a solid line and a shaded region represent an average and a 95% bootstrap confidence interval over 30 seeds in case of ED2 (ours) and 7 seeds otherwise. All of these experiments test ED2 presented in Section 3 with Algorithm 2 used for evaluation (the ensemble critic variant). We call Algorithm 2 a ’vote policy’.
  Algorithm 2 Vote policy 1: Input: ensemble size K; policy θk and Q-function φk,1 parameters where k ∈ [1, . . . ,K]; max
  action scale M ; 2: function VOTE_POLICY(s, c)
  ak = M tanh (µθk(s)) for k ∈ [1, . . . ,K] (1)
  3: if use arbitrary critic then
  qk = Qφc,1(s, ak) for k ∈ [1, . . . ,K] (2)
  4: else use ensemble critic
  qk = 1
  K ∑ i∈[1...K] Qφi,1(s, ak) for k ∈ [1, . . . ,K] (3)
  5: return ak for k = argmaxk qk
sections.heading: B.1 ENSEMBLE
sections.text: Prior networks We tested if our algorithm can benefit from prior networks (Osband et al., 2018). It turned out that the results are very similar on OpenAI Gym MuJoCo tasks, see Figure 14. However, the prior networks are useful on our crafted hard-exploration Humanoid environments, see Figure 15.
  Ensemble size Figure 16 shows ED2 with different ensemble sizes. As can be seen, the ensemble of size 5 (which we use in ED2) achieves good results, striking a balance between performance and computational overhead.
  Data bootstrap Osband et al. (2016) and Lee et al. (2020) remark that training an ensemble of agents using the same training data but with different initialization achieves, in most cases, better performance than applying different training samples to each agent. We confirm this observation in Figure 17. Data bootstrap assigned each transition to each agent in the ensemble with 50% probability.
  SOP bigger networks and training intensity We checked if simply training SOP with bigger networks or with higher training intensity (a number of updates made for each collected transition) can get it close to the ED2 results. Figure 18 compares ED2 to SOP with different network sizes, while Figure 19 compares ED2 to SOP with one or five updates per environment step. It turns out that bigger networks or higher training intensity does not improve SOP performance.
sections.heading: B.2 EXPLORATION
sections.text: Vote policy In this experiment, we used the so-called "vote policy" described in Algorithm 2. We use it for action selection in step 5 of Algorithm 3 in two variations: (1) where the random critic, chosen for the duration of one episode, evaluates each actor’s action or (2) with the full ensemble of critics for actors actions evaluation. Figure 20 shows that the arbitrary critic is not much different from our method. However, in the case of the ensemble critic, we observe a significant performance drop suggesting deficient exploration.
  UCB We tested the UCB exploration method from Lee et al. (2020). This method defines an upper-confidence bound (UCB) based on the mean and variance of Q-functions in an ensemble and selects actions with the highest UCB for efficient exploration. Figure 21 shows that the UCB exploration method makes the results of our algorithm worse.
  Gaussian noise While our method uses ensemble-based temporally coherent exploration, the most popular choice of exploration is injecting i.i.d. noise (Fujimoto et al., 2018; Wang et al., 2020). We evaluate if these two approaches can be used together. We used Gaussian noise with the standard deviation of 0.29, it is the default value in Wang et al. (2020). We found that the effects are taskspecific, barely visible for Hopper and Walker, positive in the case of Humanoid, and negative for Ant – see Figure 22. In a more refined experiment, we varied the noise level. With more noise the Humanoid results are better, whereas the And results are worse – see Figure 23.
sections.heading: B.3 EXPLOITATION
sections.text: We used the vote policy, see Algorithm 2, as the evaluation policy in step 21 of Algorithm 3. Figure 24 shows that the vote policy does worse on the OpenAI Gym MuJoCo tasks. However, on our custom Humanoid tasks introduced in Section 4, it improves our agent performance – see Figure 25.
sections.heading: B.4 NORMALIZATION
sections.text: We validated if rewards or observations normalization (Andrychowicz et al., 2020a) help our method. In both cases, we keep the empirical mean and standard deviation of each reward/observation coordinate, based on all rewards/observations seen so far, and normalize rewards/observations by subtracting the empirical mean and dividing by the standard deviation. It turned out that only the observations normalization significantly helps the agent on Humanoid, see Figures 26 and 27. The action normalization influence is tested in Appendix C.
  B.5 Q-FUNCTION UPDATES
  Huber loss We tried using the Huber loss for the Q-function training. It makes the results on all tasks worse, see Figure 28.
sections.heading: C ABLATION STUDY
sections.text: In this section, we ablate the ED2 components to see their impact on performance and stability. We start with the ensemble exploration and exploitation and then move on to the action normalization and the ERE replay buffer. In all plots, a solid line and a shaded region represent an average and a 95% bootstrap confidence interval over 30 seeds in all but action normalization and ERE replay buffer experiments, where we run 7 seeds.
  Exploration & Exploitation In the first experiment we wanted to isolate the effect of ensemblebased temporally coherent exploration on the performance and stability of ED2. Figures 29-32 compare the performance and stability of ED2 and one baseline, SOP, to ED2 with the single actor (the first one) used for evaluation in step 21 of Algorithm 3. It is worth noting that the action selection during the data collection, step 5 in Algorithm 3, is left unchanged – the ensemble of actors is used for exploration and each actor is trained on all the data. This should isolate the effect of exploration on
  the test performance of every actor. The results show that the performance improvement and stability of ED2 does not come solely from the efficient exploration. ED2 ablation performs comparably to the baseline and is even less stable.
  In the next experiment, we wanted to check if the ensemble evaluation is all we need in that event. Figure 33 compares the performance of ED2 and one baseline, SOP, to ED2 with the single actor (the first one) used for the data collection in step 5 of Algorithm 3. The action selection during the evaluation, step 21 in Algorithm 3, is left unchanged – the ensemble of actors is trained on the data collected only by one of the actors. We add Gaussian noise to the single actor’s actions for exploration as described in Appendix B.2. The results show that the ensemble actor test performance collapses, possibly because of training on the out of distribution data. This implies that the ensemble of actors, used for evaluation, improves the test performance and stability. However, it is required that the same ensemble of actors is also used for exploration, during the data collection.
  Action normalization The implementation details of the action normalization are described in Appendix E. Figure 34 shows that the action normalization is especially required on the Ant and Humanoid environments, while not disrupting the training on the other tasks.
  ERE replay buffer The implementation details of the ERE replay buffer are described in Appendix E. In Figure 35 we observe that it improves the final performance of ED2 on all tasks, especially on Walker2d and Humanoid.
sections.heading: D EXPERIMENTAL SETUP
sections.text: Plots In all evaluations, we used 30 evaluation episodes to better access the average performance of each policy, as described in Section 2. For a more pleasant look and easier visual assessment, we smoothed the lines using an exponential moving average with a smoothing factor equal 0.4.
  OpenAI Gym MuJoCo In MuJoCo environments, that we used, a state is defined by (x, y, z) position and velocity of the robot’s root, and angular position and velocity of each of its joints. The observation holds almost all information from the state except the x and y position of the robot’s root. The action is a torque that should be applied to each joint of the robot. Sizes of those spaces for each environment are summarised in Table 2.
  MuJoCo is a deterministic physics engine thus all simulations conducted inside it are deterministic. This includes simulations of our environments. However, to simplify the process of data gathering and to counteract over-fitting the authors of OpenAI Gym decided to introduce some stochasticity. Each episode starts from a slightly different state - initial positions and velocities are perturbed with random noise (uniform or normal depending on the particular environment).
  E IMPLEMENTATION DETAILS
  Architecture and hyper-parameters In our experiments, we use deep neural networks with two hidden layers, each of them with 256 units. All of the networks use ReLU as an activation, except on the final output layer, where the activation used varies depending on the model: critic networks use no activation, while actor networks use tanh() multiplied by the max action scale. Table 3 shows the hyper-parameters used for the tested algorithms.
  Action normalization Our algorithm employs action normalization proposed by Wang et al. (2020). It means that before applying the squashing function (e.g. tanh()), the outputs of each actor network are normalized in the following way: let µ = (µ1, . . . , µA) be the output of the actor’s network and let G = ∑A i=1 |µi|/A be the average magnitude of this output, where A is the action’s dimensionality. If G > 1 then we normalize the output by setting µi to µi/G for all i = 1, . . . , A. Otherwise, we leave the output unchanged. Each actor’s outputs are normalized independently from other actors in the ensemble.
  Algorithm 3 ED2 - Ensemble Deep Deterministic Policy Gradients Input: ensemble sizeK; init. policy θk andQ-functions φk,1, φk,2 param. where k ∈ [1, . . . ,K]; replay buffer D; max action scale M ; target smoothing std. dev. σ; interpolation factor ρ;
  1: Set the target parameters φ̄k,1 ← φk,1, φ̄k,2 ← φk,2 2: Sample the current policy index c ∼ U([1, . . . ,K]). 3: Reset the environment and observe the state s. 4: repeat 5: Execute action a = M tanh (µθc(s)) . µ uses the action normalization 6: Observe and store (s, a, r, s′, d) in the replay buffer D. 7: Set s← s′ 8: if episode is finished then 9: Reset the environment and observe initial state s. 10: Sample the current policy index c ∼ U([1, . . . ,K]). 11: if time to update then 12: for as many as steps done in the environment do 13: Sample a batch of transitions B = {(s, a, r, s′, d)} ⊂ D . uses ERE 14: Compute targets
  yk(r, s ′, d) = r + γ(1− d) min
  i=1,2 Qφ̄k,i(s
  ′, a′k)
  a′k = M tanh (µθk(s ′) + ) , ∼ N (0, σ)
  15: Update the Q-functions by one step of gradient descent using
  ∇φk,i 1 |B| ·K ∑
  (s,a,r,s′,d)∈B
  ( Qφk,i(s, a)− yk(r, s′, d) )2 for i ∈ {1, 2}, k ∈ [1, . . . ,K]
  16: Update the policies by one step of gradient ascent using
  ∇θk 1 |B| ·K ∑ s∈B Qφk,1(s, µθk(s)) for k ∈ [1, . . . ,K]
  17: Update target parameters with
  φ̄k,i ← ρφ̄k,i + (1− ρ)φk,i for i ∈ {1, 2}, k ∈ [1, . . . ,K]
  18: if time to evaluate then 19: for specified number of evaluation runs do 20: Reset the environment and observe the state s. 21: Execute policy a = 1K ∑K i=1M tanh (µθi(s)) until the terminal state. 22: Record and log the return. 23: until convergence
  Emphasizing Recent Experience We implement the Emphasizing Recent Experience (ERE) mechanism from Wang et al. (2020). ERE samples non-uniformly from the most recent experiences stored in the replay buffer. Let B be the number of mini-batch updates and |D| be the size of the replay buffer. When performing the gradient updates, we sample from the most recent cb data points stored in the replay buffer, where cb = |D| · ηb 1000 B for b = 1, . . . , B.
  The hyper-parameter η starts off with a set value of η0 and is later adapted based on the improvements in the agent training performance. Let Irecent be the improvement in terms of training episode returns made over the last |D|/2 time-steps and Imax be the maximum of such improvements over the course of the training. We adapt η according to the formula:
  η = η0 · Irecent Imax + 1− Irecent Imax
  Our implementation uses the exponentially weighted moving average to store the value of Irecent. More concretely, we define Irecent based on two additional parameters Rrecent and Rprev so that Irecent = Rrecent −Rprev . Those parameters are then updated whenever we receive a new training episode return ep_ret:
  Rrecent = λrecent · ep_ret+ (1− λrecent) ·Rrecent Rprev = λprev · ep_ret+ (1− λprev) ·Rprev
  where λprev = T/b |D|2 c, λrecent = 10 · λprev and T is the maximum length of an episode.
  Hardware During the training of our models, we employ only CPUs using a cluster where each node has 28 available cores of 2.6 GHz, alongside at least 64 GB of memory. The running time of a typical experiment did not exceed 24 hours.