summaries: The paper studies the problem of advantage estimation for actor-critic RL algorithms. The key observation is that the advantage can be computed using 1-step returns, 2-step returns, etc. The paper suggests that, instead of choosing a fixed n, we should aggregate these advantageous together. If the maximum is taken, the resulting policy will be exploratory (i.e., have a "promotion focus"); if the minimum advantage is taken, the resulting policy will be risk sensitive (i.e., have a "regulatory focus"). This paper focuses on risk-aware reinforcement learning, where an agent could be encouraged to take more risk (high reward, high variance states) or avoid risk (low variance states). Risk control is instantiated by different ways of estimating the advantage of a state (max/min instead of average). Experiments on several environments show good performance of the proposed algorithm. This paper presents a modification to policy gradient methods that are computed from advantage function estimates. For a given trajectory of n steps, there are n different advantage function estimates: from 1-step to n-step. GAE (Schulman, 2016) proposes to take an exponentially weighted average of these estimates to compute the policy gradients. This paper proposes instead to use order statistics to compute the policy gradient; e.g. the most optimistic estimate, the most pessimistic estimate, or the most extreme estimate.  The paper introduces a regulatory ratio: the probability of using the averaged advantage estimate vs using the order statistic, for computing the policy gradients. This hyper-parameter is justified on the optimistic case (max advantage), as a way to prevent overtly optimistic estimates. The paper conducts experiments on  different domains (sparse and dense rewards, discrete and continuous actions, fully observable and partially observable environments) which show the effect of choosing different order statistics and regulatory ratio on the policy performance.
abstractText: The estimation of advantage is crucial for a number of reinforcement learning algorithms, as it directly influences the choices of future paths. In this work, we propose a family of estimates based on the order statistics over the path ensemble, which allows one to flexibly drive the learning process in a promotion focus or prevention focus. On top of this formulation, we systematically study the impacts of different regulatory focuses. Our findings reveal that regulatory focus, when chosen appropriately, can result in significant benefits. In particular, for the environments with sparse rewards, promotion focus would lead to more efficient exploration of the policy space; while for those where individual actions can have critical impacts, prevention focus is preferable. On various benchmarks, including MuJoCo continuous control, Terrain locomotion, Atari games, and sparse-reward environments, the proposed schemes consistently demonstrate improvement over mainstream methods, not only accelerating the learning process but also obtaining substantial performance gains.
references.authors: John William Atkinson
references.title: An introduction to motivation
references.authors: Gabriel Barth-Maron Matthew W Hoffman David Budden Will Dabney Dan Horgan Alistair Muldal Nicolas Heess Timothy Lillicrap
references.title: Distributed distributional deterministic policy gradients
references.venue: arXiv preprint arXiv:1804.08617,
references.authors: M.G. Bellemare Y. Naddaf J. Veness M. Bowling
references.title: The arcade learning environment: An evaluation platform for general agents
references.venue: Journal of Artificial Intelligence Research,
references.authors: Marc G Bellemare Will Dabney Rémi Munos
references.title: A distributional perspective on reinforcement learning
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Yinlam Chow Mohammad Ghavamzadeh
references.title: Algorithms for cvar optimization in mdps
references.venue: In Advances in neural information processing systems,
references.authors: Yinlam Chow Aviv Tamar Shie Mannor Marco Pavone
references.title: Risk-sensitive and robust decisionmaking: a cvar optimization approach
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Ellen Crowe E Tory Higgins
references.title: Regulatory focus and strategic inclinations: Promotion and prevention in decision-making
references.venue: Organizational behavior and human decision processes,
references.authors: Will Dabney Georg Ostrovski David Silver Remi Munos
references.title: Implicit quantile networks for distributional reinforcement learning
references.venue: In International Conference on Machine Learning,
references.authors: Erick Delage Shie Mannor
references.title: Percentile optimization in uncertain markov decision processes with application to efficient exploration
references.venue: In Proceedings of the 24th international conference on Machine learning,
references.authors: Erick Delage Shie Mannor
references.title: Percentile optimization for markov decision processes with parameter uncertainty
references.venue: Operations research,
references.authors: Justin Fu John Co-Reyes Sergey Levine
references.title: Ex2: Exploration with exemplar models for deep reinforcement learning
references.venue: Advances in Neural Information Processing Systems
references.authors: Scott Fujimoto Herke van Hoof David Meger
references.title: Addressing function approximation error in actor-critic methods
references.venue: arXiv preprint arXiv:1802.09477,
references.authors: Jeffrey A Gray
references.title: Précis of the neuropsychology of anxiety: An enquiry into the functions of the septo-hippocampal system
references.venue: Behavioral and Brain Sciences,
references.authors: Nicolas Heess Srinivasan Sriram Jay Lemmon Josh Merel Greg Wayne Yuval Tassa Tom Erez Ziyu Wang SM Eslami Martin Riedmiller
references.title: Emergence of locomotion behaviours in rich environments
references.venue: arXiv preprint arXiv:1707.02286,
references.authors: E Tory Higgins
references.title: Beyond pleasure and pain
references.venue: American psychologist,
references.authors: E Tory Higgins
references.title: Making a good decision: value from fit
references.venue: American psychologist,
references.authors: E Tory Higgins Lorraine Chen Idson Antonio L Freitas Scott Spiegel Daniel C Molden
references.title: Transfer of value from fit
references.venue: Journal of personality and social psychology,
references.authors: Ronald A Howard James E Matheson
references.title: Risk-sensitive markov decision processes
references.venue: Management science,
references.authors: Angela Y Lee Jennifer L Aaker
references.title: Bringing the frame into focus: the influence of regulatory fit on processing fluency and persuasion
references.venue: Journal of personality and social psychology,
references.authors: Shie Mannor Ofir Mebel Huan Xu
references.title: Lightning does not strike twice: robust mdps with coupled uncertainty
references.venue: In Proceedings of the 29th International Coference on International Conference on Machine Learning,
references.authors: Volodymyr Mnih Adria Puigdomenech Badia Mehdi Mirza Alex Graves Timothy Lillicrap Tim Harley David Silver Koray Kavukcuoglu
references.title: Asynchronous methods for deep reinforcement learning
references.venue: In International conference on machine learning,
references.authors: Tetsuro Morimura Masashi Sugiyama Hisashi Kashima Hirotaka Hachiya Toshiyuki Tanaka
references.title: Nonparametric return distribution approximation for reinforcement learning
references.venue: In Proceedings of the 27th International Conference on Machine Learning
references.authors: Arnab Nilim Laurent El Ghaoui
references.title: Robustness in markov decision problems with uncertain transition matrices
references.venue: In Advances in Neural Information Processing Systems,
references.authors: LA Prashanth
references.title: Policy gradients for cvar-constrained mdps
references.venue: In International Conference on Algorithmic Learning Theory,
references.authors: LA Prashanth Mohammad Ghavamzadeh
references.title: Actor-critic algorithms for risk-sensitive mdps
references.venue: In Advances in neural information processing systems,
references.authors: John Schulman Sergey Levine Pieter Abbeel Michael Jordan Philipp Moritz
references.title: Trust region policy optimization
references.venue: In International Conference on Machine Learning,
references.authors: John Schulman Philipp Moritz Sergey Levine Michael Jordan Pieter Abbeel
references.title: High-dimensional continuous control using generalized advantage estimation
references.venue: arXiv preprint arXiv:1506.02438,
references.authors: John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov
references.title: Proximal policy optimization algorithms
references.venue: arXiv preprint arXiv:1707.06347,
references.authors: Harm Seijen Rich Sutton
references.title: True online td (lambda)
references.venue: In International Conference on Machine Learning,
references.authors: Elena Smirnova Elvis Dohmatob Jérémie Mary
references.title: Distributionally robust reinforcement learning
references.venue: arXiv preprint arXiv:1902.08708,
references.authors: Richard S Sutton Andrew G Barto
references.title: Reinforcement learning: An introduction
references.authors: Aviv Tamar Dotan Di Castro Shie Mannor
references.title: Policy gradients with variance related risk criteria
references.venue: In Proceedings of the 29th International Coference on International Conference on Machine Learning,
references.authors: Aviv Tamar Yonatan Glassner Shie Mannor
references.title: Optimizing the cvar via sampling
references.venue: In Twenty-Ninth AAAI Conference on Artificial Intelligence,
references.authors: Hado van Hasselt
references.title: Reinforcement Learning in Continuous State and Action Spaces, pp. 207–251
references.venue: ISBN 978-3-642-27645-3
references.authors: Hado Van Hasselt Marco A Wiering
references.title: Reinforcement learning in continuous action spaces
references.venue: IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning,
references.authors: Oriol Vinyals Timo Ewalds Sergey Bartunov Petko Georgiev Alexander Sasha Vezhnevets Michelle Yeo Alireza Makhzani Heinrich Küttler John Agapiou Julian Schrittwieser
references.title: Starcraft ii: A new challenge for reinforcement learning
references.venue: arXiv preprint arXiv:1708.04782,
references.title: Learning from delayed rewards
references.authors: Huan Xu Shie Mannor
references.title: The robustness-performance tradeoff in markov decision processes
references.venue: In Advances in Neural Information Processing Systems,
sections.heading: 1 INTRODUCTION
sections.text: It is not enough to do good; one must do it the right way.
  The best-known motivational principle that has been recognized in psychology (Gray, 1982; Atkinson, 1964) is the hedonic principle that people approach pleasure and avoid pain. However the principle does not explain how people approach and avoid. Higgins (1997) proposes the regulatory focus theory to address this question. A promotion focus is concerned with advancement, growth, and accomplishment, and a prevention focus is concerned with security, safety, and responsibility (Crowe & Higgins, 1997). The theory hypothesized that the promotion focus inclination insures against errors of omission, and the prevention focus inclination insures against errors of commission. Furthermore, the regulatory fit theory (Higgins, 2000; Higgins et al., 2003) proposes that when people pursue a goal in a manner that fit their regulatory orientation, they are stronger motivated during goal pursuit. For example, Lee & Aaker (2004) showed that ads are more persuasive if there is a regulatory fit: use gain frames (e.g. “great looks and exceptional engineering”) for individuals who are eager to approach positive outcomes, and use loss frames (e.g. “don’t be stranded with a disabled vehicle without an emergency safety kit”) for individuals who are vigilant to avoid negative outcomes.
  We study the phenomenon of regulatory fit in the context of reinforcement learning (RL). Be aware that humans own real emotions and are fundamentally more complex than RL agents, thus we borrow terminologies in psychology while trying to retain the resemblance. We hypothesize that different RL environments are more fit with agents of different regulatory focus. For example, in the environments with sparse rewards, it might be advisable to actively exploring those policies that show potentials without being discouraged by a few failed trials; while in the environments that are fragile, e.g. those where a wrong choice of action can lead to catastrophic consequences down the path, a more conservative, risk-averse stance might be preferable. There are three questions need to answer: (1) How to automatically determine the suitable regulatory focus for an environment; (2) How to train an agent with a specified regulatory focus; (3) Is there any benefit when training the agent with the regulatory focus that fits the environment? In this article, we aim to answer the second and third questions and leave the first question open.
  The research on deep reinforcement learning is gaining momentum in recent years. A number of learning methods, such as Proximal Policy Optimization algorithm (PPO) (Schulman et al., 2017),
  Trust-Region Policy Optimization algorithm (TRPO) (Schulman et al., 2015a), and Advantage ActorCritic algorithm (A2C) (Mnih et al., 2016), have been developed. These methods and their variants have achieved great success in challenging problems, e.g. continuous control (Schulman et al., 2015b), locomotion (Heess et al., 2017), and video games (Vinyals et al., 2017). The core of all these methods is the estimation of the advantage A(st, at), i.e. the gain in the expected cumulative reward relative to the state value V (st) if a certain action at is taken. A common practice is to use the n-step estimate over the sampled trajectories. This way has been widely used in actor-critic algorithms such as the A2C. Schulman et al. presents a generalization called Generalized Advantage Estimator (GAE) (Schulman et al., 2015b), which blends 1 to n-step estimators with exponentially decayed weights, thus reducing the variance of policy gradients. For convenience, we refer to the set of 1 to n-step estimators along a trajectory as the path ensemble.
  We propose to take the maximum over path ensemble to implement the promotion focus inclination, and take the minimum over path ensemble to implement to prevention inclination during training. Intuitively, when taking the maximum or minimum, the agent is comparing a list of counterfactual speculations. When it looks at a k-step estimation where k ≤ n, it is thinking what if it didn’t took those actions after step k? For a promotion focus, in order to insure hits and insure against omission, it would favor the estimation with largest value because it has the possibility of being that good. Similarly, for a prevention focus, it prefers the minimum to insure against errors of commission.
  We systematically studied different estimation schemes on various environments, including the sparse-reward environments (Fu et al., 2017), the Terrain RL Simulator (Berseth et al., 2018) for biped locomotion, the Atari games (Bellemare et al., 2013), and the set of MuJoCo environments for continuous control (Brockman et al., 2016). Our study shows that the proposed estimation strategies, when chosen appropriately, can substantially outperform the standard way. In particular, the promotion focus inclination, i.e. taking the maximum over path ensemble, greatly improves the learning efficiency in the environments with sparse rewards. On the other hand, the prevention focus inclination, i.e. taking the minimum, can effectively stabilize the learning by avoiding risky actions.
  It is noteworthy that the proposed strategic inclinations differ essentially from the recent two lines of work on robustness (Smirnova et al., 2019; Delage & Mannor, 2007) and risk-sensitivity (Tamar et al., 2012; Chow & Ghavamzadeh, 2014). Robust MDP optimizes for the worst case when there exist uncertainties in the parameters; risk-sensitive MDP optimizes the value of a risk measure. Our method does not modify the training objective. Instead, it controls the policy gradient towards policies with promotion or prevention inclinations through the alternative ways of estimating the advantage.
sections.heading: 2 RELATED WORK
sections.text: Estimation of Advantage and Action-Value The n-step advantage estimation is derived from the n-step return, which is used in the n-step TD or the n-step Sarsa method (Sutton & Barto, 2011). Similarly, the generalized advantage estimator (Schulman et al., 2015b) is analogous to the λ-return (Watkins, 1989) in the TD(λ) method (Seijen & Sutton, 2014). They are all linear combinations over the path ensemble. A nonlinear combination scheme is seen in the work of Twin Delayed DDPG (TD3) (Fujimoto et al., 2018), which uses the min of two critics to estimate Q. However, it aims to mitigate the overestimation problem in deep Q-learning, and the objects be combined are the network outputs instead of the estimators based on the cumulative rewards. Another nonlinear combination is the Positive Temporal Difference (PTD) (Van Hasselt & Wiering, 2007; van Hasselt, 2012), where the advantage is set as 1 when the 1-step estimate of advantage is positive, and is set to 0 otherwise. The benefit of limiting policy updates only toward those actions that have positive advantages is increasing the stability of learning.
  Robustness and Risk-Sensitivity in RL In the assumption of robust MDPs (Nilim & El Ghaoui, 2004; Xu & Mannor, 2007; Delage & Mannor, 2010; Mannor et al., 2012; Smirnova et al., 2019), the parameters of the problem lie in an uncertainty set, and the target is to find a solution that performs best under the worst cases. On the other hand, risk-sensitive MDPs consider the uncertainty of the rewards. The objective is to minimize the risk-measure, which is defined by the exponential utility (Howard & Matheson, 1972), the Conditional Value-at-Risk (CVaR) (Chow & Ghavamzadeh, 2014; Prashanth, 2014; Chow et al., 2015; Tamar et al., 2015), the percentile (Delage & Mannor, 2007) or the variance (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013) of the cumulative
  rewards. The statistics over the path ensemble is not the same as those statistics for advantage. However, using a suitable nonlinear combination of the numbers in the path ensemble, the exploration strategy in our algorithms can bias the learning toward risk-averse or risk-seeking policies. Unlike the robust and risk-sensitive approaches, which typically introduce involved computations, our approach is straightforward to implement and requires minimal changes to incorporate the idea into existing algorithms such as A2C, PPO, TRPO and their variants.
  Distributional RL The value distribution is the object of study in distributional RL. Since the full distribution of return is difficult to estimate, researchers have adopted nonparametric methods (Morimura et al., 2010) or used simple distributions (Bellemare et al., 2017; Dabney et al., 2018; Barth-Maron et al., 2018) to approximate it. The RL algorithms can be formulated with any criterion based on the distribution of return, such as the aforementioned risk-sensitivity measures. Again, the distribution formed by elements in the path ensemble does not represent the full distribution of return, but some joint distribution that we can exploit.
sections.heading: 3 REGULATORY FOCUS BY ORDER STATISTICS OVER PATH ENSEMBLE
sections.text: In this section, we first review the preliminary knowledge on RL. Then we introduce the central concept of this article: regulatory focus implemented by estimators over path ensembles. Specifically, we focus on the family of order statistics. Next, we give an illustrative study on the max statistics over the path ensemble, where we show that the induced estimator of action-values would influence the optimization process of RL algorithms. With this specific example in mind, we give a practical algorithm to incorporate the general estimators into existing RL algorithms in the last section.
sections.heading: 3.1 PRELIMINARY
sections.text: We consider the standard formulation of RL, which is typically modeled as an MDP (S,A, T , γ, R), where S is the state space, A is the action space, T = {Psa(·) | s ∈ S, a ∈ A} is the transition probabilities, γ ∈ (0, 1] is the discount factor, and R is the reward function. At timestep t, the agent in state st interacts with the environment by choosing an action at ∼ π(st) following the policy π, and receives a reward Rt from the environment. The environment then transits to the next state st+1. The discounted return is defined as Gt := ∑T−1 i=t γ
  i−tRi, and the goal of RL is to maximize the expected return J = Es0∼S0 [G0 | s0], where S0 is the distribution of initial states. The action-value function Qπ(st, at) := E [Gt | st, at] under a policy π is the expected return of taking action at ∼ π(st) in state st. In policy-based model-free deep RL algorithms, policy πθ with parameters θ is optimized via gradient ascent. An estimation of the gradient is the policy gradient
  ∇θJ(θ) = Eτ∼πθ(τ) [ T∑ t=0 ∇θπθ(at | st) πθ(at | st) A(st, at) ] , (1)
  where τ = {s0, a0, s1, a1, . . . , sT , aT } is a trajectory following the policy. A(st, at) := Q(st, at)− V (st) is the advantage function. The n-step look ahead estimation of advantage function is given by
  Â (n) t := n−1∑ i=0 ( γirt+i ) + γnV (st+n)− V (st), n = 1, 2, . . . , T − t. (2)
sections.heading: 3.2 ORDER STATISTICS OVER THE PATH ENSEMBLE
sections.text: Given a trajectory, the set of n-step estimators E := {Â(1)t , Â (2) t , . . . } is defined as the path ensemble for the pair (st, at). We have briefly mentioned that the maximum over path ensemble can achieve the promotion focus, and the minimum over path ensemble can achieve the prevention focus in Introduction. The max statistics and the min statistics over the path ensemble are respectively computed by
  Âmaxt := max i
  { Â
  (i) t } , Âmint := min
  i
  { Â
  (i) t
  } . (3)
  For many environments that are neither sparse nor fragile, whether they are more close to promotion focus or prevention focus is unclear without in-depth inspection. They also might be too complex to
  be simply described by any one of the two regulatory types. For these cases, we propose a generalized order statistics which is a hybrid of the maximum and minimum. We call it the max-abs statistics which took the element with the maximum absolute value,
  Âmax-abst := argmax A∈{Â(i)t } |A|. (4)
  To understand how elements in the path ensemble relates to regulatory focus, we interpret the k-step estimation as counterfactual speculations when k ≤ n. Since k-step estimation only involves the first k states after the evaluated state, all actions and the followed observations are dropped and be replaced by the state-value at step-k. This underlies a counterfactual thinking that what if the agent didn’t do the action sequence at+k+1, at+k+2, . . ., and replace the situation by the average behavior? Then different regulatory focus would treat the set of counterfactual speculations in different ways.
  The max Statistics and the Promotion Focus A promotion focus is concerned with advancement, growth, and accomplishment. The promotion focus inclination is to insure hits and insure against errors of omission. Facing the set of counterfactual speculations, it would choose the one with the maximum value to insure any action at in st that is possibly to behave good to have more importance. The max statistics leads to optimistic estimation of advantages. It is beneficial for sparse-reward environments as actions that may lead to large returns will not be buried by the later bad actions.
  The min Statistics and the Prevention Focus A prevention focus is concerned with security, safety, and responsibility. The prevention focus inclination is to insure correct rejections and insure against errors of commission. Facing the set of counterfactual speculations, it would choose the one with the minimum value to avoid wrongly acknowledge an action that only temporarily leads to high rewards. The min statistics implies the risk-averse bias. It avoids actions that may cause bad states later. By decreasing the advantage estimation of those actions, it makes the optimization direction away from them. This property is useful for fragile environments, such as the biped locomotion environments which are sensitive to joint motions of the characters.
  The max-abs Statistics and Overreaction In the max-abs statistics, the estimator with the largest absolute value in the path ensemble is chosen. This means that an action is evaluated as either overly good when the largest positive advantage is selected, or overly bad when the smallest negative advantage is selected. We refer to this behavior as overreaction. Overreaction is not a terminology in regulatory focus theory, but a descriptive term for the phenomenon concerning the effects of the max-abs statistics. This statistics implements a heuristic that makes the good paths look better and the bad paths look worse. As will be shown in experiments, this heuristic generally improves sample efficiency for the MuJoCo and Atari environments, where their regulatory focuses are unclear.
  Both the maximum and the minimum belong to the family of order statistics, where the k-th order statistics equals to the k-th smallest value. From computational perspective, it is also viable to test whether the more general order statistics are effective for training, although lacking strong explanations. We put the study on general order statics in Appendix A.
  3.3 AN ILLUSTRATIVE STUDY ON THE max STATISTICS
  Before delving into the discussion of estimating advantages, we first look at a concrete example of how the max statistics ofQ-values affect the learning process of the policy iteration algorithm (Sutton & Barto, 2011, Section 4.3). In Figure 1(a), an MDP with 6 states is drawn. For simplicity, we assume that both the state transitions and the rewards are deterministic, and the discount factor is 1. In this example, only the two actions in state s3 can get rewards, where R(s3, a1, s5) = −2 and R(s3, a2, s6) = 2. At the initialization step of the policy iteration algorithm, the policy is initialized as the uniform random policy π1. In the first step of the policy iteration, we evaluate the Q function and value function V of this random policy π1. Under policy π1, all possible trajectories are also shown in Figure 1(a). We compute that Qπ1(s1, a1) = 0, and Qπ1(s1, a2) = 0. So in the next step, the greedy policy π2 for state s1 is still at random. We compute Q̂max using the max statistics,
  Q̂max(s1, a1) = E [ max i {R(i)} | s1, a1 ] = 0, Q̂max(s1, a2) = E [ max i {R(i)} | s1, a2 ] = 1.
  (5)
  Using Q̂max, the greedy policy at state s1 is argmaxa Q̂ max(s1, a) = a2. Actually, the algorithm with max statistics finds the optimal policy within only 1 step in this example.
  We see that the max statistics over the path ensembles converges faster in the toy example because it discovers the potentially good action a2 at state s1 earlier than using the standard way. As the computation of the second trajectory in Figure 1(a) shows, the max statistics blacklists the bad action a1 in state s3. In a trajectory, when an action is so bad that the later actions cannot compensate for the caused loss, it will be replaced by the average actions. This good-action discovering ability is a characteristic of promotion focus, and is suitable for sparse-reward environments. In the early exploration stage in the training, the agent barely receives positive signals. Many trajectories might happens to act well at some time, but then go to bad states later due to wrong actions. The max statistics can highlight actions that are possible to get high rewards in any middle step of the trajectory. By effectively discovering good actions, this method is expected to improve the sampling efficiency.
  The foundational position of advantage in actor-critic algorithms is attributed by the policy gradient theorem in Equation (1). A sample (st, at) makes the policy network πθ adjust its parameters according to the estimation of advantage Ât(st, at). The probability of πθ(st) = at is increased when the advantage is positive, and decreased when the advantage is negative. If the estimation of advantage Ât(st, at) is manipulated to be larger, the learned policy is then biased toward the action at at state st. Thus when the estimation of certain advantages Ât(st, at) is manipulated properly, the learned policy can be biased toward the desired regulatory orientation.
sections.heading: 3.4 INCORPORATING REGULATORY FOCUS INTO RL ALGORITHMS
sections.text: We should be careful that in the presence of uncertainty or noise, applying the max statistics in policy iteration may fail to improve policy. We give two examples when the estimation is overly optimistic. The first example, which is shown in Figure 1(b), is caused by the uncertainty in state transitions. We have Q̂max(s1, a1) = 0.5 and Q̂max(s1, a2) = 1, which implies that action a2 should be chosen in state s1. However, the optimal strategy is to select a1. This is caused by the the max statistics’ ignorance on the coupling risks. When taking an action, both good and bad may happen. The bad cases is ignored in the estimation. However, we cannot avoid bad next-state when selecting that action. The second example, shown in Figure 1(c), is a case when the reward owns randomness. The problem is that the maximum operation over the ensemble causes overestimation, a symptom that also troubles the Q-learning algorithm as discussed in TD3 (Fujimoto et al., 2018).
  We overcome this shortcoming from two aspects: (1) implement the regulatory focus as an exploration strategy. Namely, the estimation from path ensemble is used with a probability ρ. We call ρ as the regulatory ratio. (2) using a small ensemble size. We argue that a small ensemble size is sufficient for practical use because elements in the path ensemble are inter-related. Roughly speaking, the difference between the i-step advantage estimator Â(i)t and the j-step estimator Â (j) t in a path ensemble is discounted exponentially with the minimum step index min(i, j) of the two. In fact, let j > i, then
  Â (j) t − Â (i) t = γ i ( j−i−1∑ s=0 (γsrt+s+i) + γ j−iV (st+j)− V (st+i) ) . (6)
  The sum in the big bracket can be assumed to be bounded. Based on this observation, the values of the estimation Â(k)t would not change too much when k becomes large. In practice using a subset of the full ensemble also reduces the computational cost. For example, we used an ensemble with only four elements E = {Â(1)t , Â (16) t , Â (64) t , Â (2048) t } in the MuJoCo physics simulator experiments.
  Summarizing the above discussion, our algorithm is designed as follows.
  Algorithm 1 Regulatory Focus by Order Statistics over Path Ensembles 1: Parameters: the ensemble set E , the order statistics Âorder over E , and the regulatory ratio ρ. 2: for each iteration do 3: Collect trajectories. 4: Compute the advantage Â and the estimation Âorder. 5: Update the policy network according to PPO, A2C, or TRPO, in which Âorder is used with
  probability ρ, and Â is used otherwise. 6: Update the critic network using the estimation of Q value, which equals to V + Â. 7: end for
  Our method can be easily plugged into any actor-critic algorithms with a minimal code change and a small computational overhead.
sections.heading: 4 EXPERIMENT
sections.text: We evaluate the performance of our algorithm on four different types of problems, including the environment with sparse reward, biped locomotion, continuous control and Atari games.
sections.heading: 4.1 SPARSE REWARD AND THE PROMOTION FOCUS INCLINATION
sections.text: The amortized EX2 algorithm (Fu et al., 2017) is designed for tasks with sparse rewards. Three environments were chosen from their paper, which represent three types of problems. The order statistics over path ensemble was implemented by modifying the advantage estimation procedure of the authors’ original code, where the TRPO is used for policy optimization. The path ensemble was composed of k-step estimators where k ∈ {1, 16, 64, 4000} for Maze and Doom, and k ∈ {1, 16, 64, 5000} for SparseHalfCheetah. The numbers 4000 and 5000 originated from the respective batch sizes. The regulatory ratio for both the max and the min statistics was set to ρ = 0.5 for Maze and Doom, and ρ = 0.3 for SparseHalfCheetah. All hyper-parameters followed the settings of the amortized EX2. The result is shown in Figure 2.
  2D Maze This environment provides a sparse reward function, where the agent can only get the reward when it is within a small radius of the goal. From the figure, we observed that the max statistics started to gain rewards at the very early stage, which indicates the promotion focus provides a supreme sample efficiency over other methods. After 500 updates, the average episodic reward of max was much higher than that of the EX2 algorithm. The min statistics was worse than EX2.
  Doom MyWayHome is a vision-based maze navigation benchmark, where the agent is required to navigate through a series of interconnected rooms before reaching the goal. A +1 reward is given for reaching the goal before timeout. The task is challenging because the inputs are realistic images in the first-person perspective. The max statistics was very effective for this challenge and it reached nearly 100% success rate. In comparison, EX2 only got an average of 0.6 episode rewards.
  SparseHalfCheetah It is a challenging continuous control task with sparse reward. Our promotion focus significantly improved the sample efficiency, and the reward after 1000 updates was 40% higher than the EX2 algorithm. Also, the min statistics was less effective than EX2 in all three benchmarks, which demonstrates that prevention focus is not effective in sparse environments.
sections.heading: 4.2 BIPED LOCOMOTION AND THE PREVENTION FOCUS INCLINATION
sections.text: The prevention focus is very useful for environments that are fragile to actions, i.e., one wrong action would lead to catastrophic results later on. The Terrain RL simulator (Berseth et al., 2018) provides such environments. Two biped walking tasks were selected, where a character needs to learn how to walk steadily forward in two different terrains, the flat and the incline, as shown in Figure 3. The character continuously receives rewards until it fells down or is stuck, in which case the episode terminates. This task is challenging because the observation only contains the pose of the character, which forces the character to learn how to walk without memorizing actions based on its location. In contrast, the locomotion tasks in MuJoCo environments contain absolute world coordinates in the observation. The action space is 11-dimensional, which corresponds to the joints of the character. The environment is fragile. If a wrong action is performed, the character might lose balance and fall down. We used PPO in this experiment. The hyper-parameters were borrowed from those designed for MuJoCo environments in the baselines’ (Dhariwal et al., 2017) PPO implementation and are also listed in Appendix. The min and max statistics were implemented on top of the PPO algorithm. The path ensemble consisted of k-step estimators where k ∈ {1, 16, 64, 2048}, and the regulatory ratio is ρ = 0.3. The results are shown in Figure 3. The prevention focus inclination via the min statistics successfully mastered the task while the vanilla PPO algorithm failed. The promotion focus inclination by the max was not very effective in these environments as the low reward curves indicate that the character fells down at the beginning of the episode. We conclude that prevention focus by the min statistics of the path ensemble helps the agent to learn in fragile environments.
sections.heading: 4.3 CONTINUOUS CONTROL AND THE OVERREACTION INCLINATION
sections.text: We tested various advantage estimations on 5 continuous control benchmarks based on the MuJoCo physics simulator. They are not sparse-reward environments, and most of them are also not sensitive to individual actions. For example, the design of the HalfCheetah agent makes it seldom fall down or be stuck. We used PPO in this set of experiments. The implementation was based on the baselines (Dhariwal et al., 2017) code repository, and the default hyper-parameters were adopted. We tested three order statistics including the max, the min and the max-abs statistics. In all settings, the path ensemble had index set {1, 16, 64, 2048}, and the regulatory ratio was set to ρ = 0.4. Results are shown in Figure 4. We observed that the max-abs statistics was consistently better than the baseline PPO algorithm, while the performances of the max and the min statistics depended upon the specific environment. So we conclude that the overreaction inclination by the max-abs statistics is generally effective for a wide range of environments.
sections.heading: 4.4 ATARI GAMES AND THE OVERREACTION INCLINATION
sections.text: The new advantage estimations were also tested on a subset of Atari game environments. We used the A2C algorithm with 4 paralleled threads. The implementation was based on the A2C codes in baselines (Dhariwal et al., 2017). Hyper-parameters were the defaults for Atari games. Since the sampled trajectory length was 5 in the default setting, the path ensemble had fewer elements, which was composed of {1, 3, 5}. The maximum trajectory length of 5 in a batch was too small to gather enough data on k-step estimations when k > 1. For example, only the first state has a valid 5-step advantage estimation in a length-5 trajectory. This affects the power of the path ensemble. To circumvent this limitation, we collected paths of length 5n, and then computed the advantage estimators using these longer trajectories. Since the batch size was n times of the original setting, it was split into n mini-batches. For Seaquest and Breakout, n = 20; for Pong and Qbert, n = 10; and for BeamRider, n = 60. The result is shown in Figure 5. In these environments, the sparsity and fragility were unknown. Obviously, the overreaction inclination generally improves the performance, whereas the max and the min statistics are only effective for a subset of environments.
sections.heading: 5 CONCLUSION
sections.text: According to the regulatory fit theory in psychology, when people engage in goal pursuit activities in a manner that fits their regulatory orientation, they feel right about what they are doing. In this paper, we proposed a simple yet effective way of exploration using the idea of regulatory fit. The regulatory focus is implemented in the context of reinforcement learning via the order statistics over path ensembles, which formed nonlinear combinations of different n-step advantage estimators. The maximum, the minimum, and the element with the maximum absolute value were studied in detail. We incorporated these advantage estimators into three widely used actor-critic algorithms including A2C, TRPO and PPO. When the promotion or prevention focus inclination fits the environment, the proposed algorithm could be effective in solving sparse-reward environments, or the fragile environments which are sensitive to individual actions. We verified the effectiveness of our approach by extensive experiments on various domains, including the continuous control, locomotion, video games, and sparse-reward environments.
sections.heading: A ABLATION STUDY
sections.text: In this section we study the behavior of general order statistics on MuJoCo environments, and then discuss the sensitivity of the algorithms to hyper-parameters.
  The General Order Statistics As shown in Figure 6, in three (InvertedDoublePendulum, Ant, and Humanoid) of the five environments, the max and min were better than the 2nd and the 3rd order statistics. This means that some environments welcome both optimistic and risk-averse exploration. Overall, the intermediate order statistics was better than the baseline, which used the GAE estimator.
  Sensitivity on the Regulatory Ratio Generally speaking, the regulatory ratio ρ had a great influence on the performance. As shown in Figure 6, the range [0.2, 0.4] is a plausible choice for most experiments. If the ratio is further increased, the performance may degrade. This is consistent with our analysis, and shows the necessity of using regulatory focus as an exploration strategy.
  Effect of Ensemble Size We built another path ensemble whose index set consisted of 12 elements. They were {1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048}. Final performances of the three order statistics, max, min and max-abs, were tested under the regulatory ratio ρ = 0.4. For the ensemble with 4 elements {1, 16, 64, 2048}, the average episode reward in the end of the training was 928, 927, 974, respectively; and for the ensemble of size 12, the numbers were 767, 854 and 867. It shows that a small ensemble size can already get most of the performance boost.
sections.heading: B COMPARE WITH -GREEDY EXPLORATION
sections.text: As an exploration method, it is helpful to compare the regulatory focus exploration with the basic -greedy exploration. The results are shown on Mujoco environments in Figure 7. We can see that completely random exploration, even in small proportion, makes the results worse.
sections.heading: C COMPARE WITH DIFFERENT CLIP THRESHOLD OF PPO
sections.text: The clipping threshold of PPO also affects the estimation of advantage. A smaller clipping threshold would result in a more conservative policy iteration. We compare the effect of smaller clipping threshold with the min statistics over path ensemble on the biped locomotion task. Results are shown in Figure 8. Clipping is adding a global minimum constraint, meaning that all trajectories share the same clipping constant. In contrast, the min of path ensemble gives prevention focus inclination in a momentary manner, i.e. per trajectory. The performance of min path ensemble is much better.
sections.heading: D PATH ENSEMBLE OF GAES
sections.text: The generalized advantage estimator GAE(γ, λ) is an exponentially-weighted average of different n-step estimators,
  GAE(γ, λ) := (1− λ) ( Â
  (1) t + λÂ (2) t + λ 2Â (3) t + · · ·
  ) . (7)
  With different hyper-parameter λ, the estimator focus on different time ranges. For example, if λ is small, then the decay would be very fast and effectively only the first steps in the trajectory is considered; if λ is close to 1, then GAE takes longer time ranges into account. Having the above observation, we can use the path ensemble of GAEs with different λs as an alternate version of the path ensemble of n-step advantages. The GAE formulation would have a smaller variance and may be beneficial for some environments. We tested the formulation of ensembling GAEs on the Mujoco environments and found substantial improvements on the Ant-v2 and Humanoid-v2 environments. Note also that on HalfCheetah-v2 and Walker2d-v2, performance drops compared to ensembling n-step advantages. The results are drawn in Figure 9.
sections.heading: E DETAILED EXPERIMENT SETTINGS
sections.text: The detailed hyper-parameter settings of the experiments are listed in Table 1.