summaries: This paper considers a generalization of the policy gradient method to optimize for arbitrary utility functions with weightings that depend on the entire CDF (rather than the expected reward). This generalization has two aspects: (1) a utility function on top of the trajectory reward and (2) a weighting function for the CDF of the trajectory reward with respect to which the expectation is performed.  The paper derives an expression for the policy gradient and also generalizes the standard variance reduction baselines. Inspired by the PPO loss, the authors then propose a clipped version of the policy gradient calling it C3PO and evaluate this on some benchmarks from the OpenAI Safety Gym, where it is found that the conservative weightings can offer improvements over the standard formulation. Risk objectives have long been investigated in reinforcement learning (RL). Most of the focus has been on classic risk measures, like exponential utility, value-at-risk (VaR), conditional value-at-risk (CVaR), leaving out, however, the cumulative prospect theory (CPT) developed by Tversky and Nobel Prize Kahneman in 1992, which has not yet been considered.  The advantage of CPT is to better model human decision-making, still allowing a wide class of risk measures, based on the utility $u$ and weighting $\omega$.  Hence, the authors consider a new risk-aware objective. Following some derivation, they compute a sample-based estimation of the gradient of this new objective w.r.t. the policy parameters.  The authors propose a PPO-like algorithm (called C3PO), which incorporates the new, risk-aware, gradient estimator.  They perform an empirical analysis on some tasks of "Safety Gym", showing that proper risk-awareness helps increase the performance of classic PPO. The paper presents an alternate approach for distributional DRL via proposing an objective inspired from Cumulative Prospect Theory (CPT, Tversky and Kahneman, 1992). They use this distributional objective in conjunction with policy gradient methodology to propose a distribution policy gradient method for risk-sensitive RL. Under their approach, the distribution of the returns is optimized to maximize some chosen function of its CDF. They experiment with different such possible distributional objectives (risk profiles) on the OpenAI Safety Gym environments and show that their approach performs better than PPO.  The article propose a policy gradient method for optimizing a CDF based criterion, inspired by CPT.  By varying the weighting function inside the objective, it is possible to change the risk-aversion of the agent. The authors derives the policy gradient for the aforementioned objective and propose an estimation technique for it. Then, they propose an algorithm which extends PPO for optimizing their objective. Empirical analysis is carried on to evaluate the approach on some modified Safety Gym environments, in which a fixed negative rewards corresponded to adverse events. The authors evaluate different objectives obtained by employing a Wang weighting function, with different values of the parameter $\eta$. They show that optimizing a cautious (or risk-averse) objective allow to obtain better results in terms of average reward w.r.t. optimizing an aggressive (or risk-seeking) one. By further exploring the parameter space, the author demonstrate that some risk-averse values of the parameters allow to outperform also the risk-neutral version of PPO w.r.t. the average reward objective.
abstractText: Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration. It also wastes an opportunity for the agent to modulate behavior based on distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose a more direct approach, whereby the distribution of full-episode outcomes is optimized to maximize a chosen function of its cumulative distribution function (CDF). This technique allows for outcomes to be weighed based on relative quality, does not require modification of the reward function to modulate agent behavior, and may be used for both continuous and discrete action spaces. We show how to achieve an asymptotically consistent estimate of the policy gradient for a broad class of CDF-based objectives via sampling, subsequently incorporating variance reduction measures to facilitate effective on-policy learning. We use the resulting algorithm to train agents with different “risk profiles” in penalty-based formulations of six OpenAI Safety Gym environments, observing that moderate emphasis on improvement in training scenarios where the agent performs poorly both increases the accumulation of positive rewards and decreases the frequency of incurred penalties. We found that, in all environments tested, the same risk profile can be used to produce both stronger overall performance than standard Proximal Policy Optimization (PPO) and higher levels of positive reward than PPO constrained by Lagrangians to maintain the same cost levels.
references.authors: Josh Achiam
references.title: Openai spinning up: Proof for don’t let the past distract you. http://spinningup
references.authors: Joshua Achiam David Held Aviv Tamar Pieter Abbeel
references.title: Constrained policy optimization
references.venue: In Proceedings of the 34th International Conference on Machine Learning (ICML),
references.authors: Dario Amodei Chris Olah Jacob Steinhardt Paul Christiano John Schulman Dan Mané
references.title: Concrete problems in AI safety
references.authors: M.G. Bellemare Y. Naddaf J. Veness M. Bowling
references.title: The arcade learning environment: An evaluation platform for general agents
references.venue: Journal of Artificial Intelligence Research,
references.authors: Marc G. Bellemare Will Dabney Rémi Munos
references.title: A distributional perspective on reinforcement learning
references.venue: In Proceedings of the 34th International Conference on Machine Learning (ICML),
references.authors: Yinlam Chow Ofir Nachum Aleksandra Faust Mohammad Ghavamzadeh Edgar A
references.title: DuéñezGuzmán. Lyapunov-based safe policy optimization for continuous control
references.venue: CoRR, abs/1901.10031,
references.authors: Will Dabney Georg Ostrovski David Silver Rémi Munos
references.title: Implicit quantile networks for distributional reinforcement learning
references.venue: In Proceedings of the 35th International Conference on Machine Learning (ICML),
references.authors: Will Dabney Mark Rowland Marc G. Bellemare Rémi Munos
references.title: Distributional reinforcement learning with quantile regression
references.venue: In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI),
references.authors: Jingliang Duan Yang Guan Shengbo Eben Li Yangang Ren Qi Sun Bo Cheng
references.title: Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors
references.venue: IEEE Transactions on Neural Networks and Learning Systems,
references.authors: Javier Garcı́a Fernando Fernández
references.title: A comprehensive survey on safe reinforcement learning
references.venue: Journal of Machine Learning Research,
references.authors: Daniel Kahneman Amos Tversky
references.title: Prospect theory: An analysis of decision under risk
references.authors: Dickson H. Leavens
references.title: Diversification of investments
references.venue: Trusts and Estates,
references.authors: Xiaoteng Ma Li Xia Zhengyuan Zhou Jun Yang Qianchuan Zhao
references.title: DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning
references.authors: German I. Parisi Ronald Kemker Jose L. Part Christopher Kanan Stefan Wermter
references.title: Continual lifelong learning with neural networks: A review
references.venue: Neural Networks,
references.authors: Lerrel Pinto James Davidson Rahul Sukthankar Abhinav Gupta
references.title: Robust adversarial reinforcement learning
references.venue: In Proceedings of the 34th International Conference on Machine Learning (ICML),
references.authors: L.A. Prashanth Cheng Jie Michael C. Fu Steven I. Marcus Csaba Szepesvári
references.title: Cumulative prospect theory meets reinforcement learning: Prediction and control
references.venue: In Proceedings of the 33nd International Conference on Machine Learning (ICML),
references.authors: John W. Pratt
references.title: Risk aversion in the small and in the large
references.venue: Econometrica, 32:122–136,
references.authors: Alex Ray Joshua Achiam Dario Amodei
references.title: Benchmarking safe exploration in deep reinforcement learning
references.authors: R. Tyrrell Rockafellar Stanislav Uryasev
references.title: Optimization of conditional value-at-risk
references.venue: Journal of Risk,
references.authors: John Schulman Philipp Moritz Sergey Levine Michael Jordan Pieter Abbeel
references.title: High-dimensional continuous control using generalized advantage estimation
references.venue: In Proceedings of the International Conference on Learning Representations (ICLR),
references.authors: John Schulman Sergey Levine Philipp Moritz Michael I. Jordan Pieter Abbeel
references.title: Trust region policy optimization
references.venue: In Proceedings of the 32nd International Conference on Machine Learning (ICML),
references.authors: John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov
references.title: Proximal policy optimization algorithms
references.authors: Aviv Tamar Yinlam Chow Mohammad Ghavamzadeh Shie Mannor
references.title: Policy gradient for coherent risk measures
references.venue: In Proceedings of the 28th International Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Amos Tversky Daniel Kahneman
references.title: Advances in prospect theory: Cumulative representation of uncertainty
references.venue: Journal of Risk and Uncertainty,
references.authors: Shaun S. Wang
references.title: A class of distortion operators for pricing financial and insurance risks
references.venue: The Journal of Risk and Insurance,
references.authors: Ronald J. Williams
references.title: Simple statistical gradient-following algorithms for connectionist reinforcement learning
references.venue: Machine Learning,
references.authors: Congbin Wu Yuanlie Lin
references.title: Minimizing risk models in Markov decision processes with policies depending on target values
references.venue: Journal of Mathematical Analysis and Applications,
references.authors: Huan Zhang Hongge Chen Chaowei Xiao Bo Li Mingyan Liu Duane Boning Cho-Jui Hsieh
references.title: Robust deep reinforcement learning against adversarial perturbations on state observations
references.venue: Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Junyu Zhang Amrit Singh Bedi Mengdi Wang Alec Koppel
references.title: Cautious reinforcement learning via distributional risk in the dual domain
references.venue: IEEE Journal on Selected Areas in Information Theory,
references.authors: Han Zhong Ethan X. Fang Zhuoran Yang Zhaoran Wang
references.title: Risk-sensitive deep RL: Varianceconstrained actor-critic provably finds globally optimal policy
sections.text: Standard deep reinforcement learning (DRL) agents aim to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued differently and outlying outcomes are given increased consideration. It also wastes an opportunity for the agent to modulate behavior based on distributional context. Several approaches to distributional DRL have been investigated, with one popular strategy being to evaluate the projected distribution of returns for possible actions. We propose a more direct approach, whereby the distribution of full-episode outcomes is optimized to maximize a chosen function of its cumulative distribution function (CDF). This technique allows for outcomes to be weighed based on relative quality, does not require modification of the reward function to modulate agent behavior, and may be used for both continuous and discrete action spaces. We show how to achieve an asymptotically consistent estimate of the policy gradient for a broad class of CDF-based objectives via sampling, subsequently incorporating variance reduction measures to facilitate effective on-policy learning. We use the resulting algorithm to train agents with different “risk profiles” in penalty-based formulations of six OpenAI Safety Gym environments, observing that moderate emphasis on improvement in training scenarios where the agent performs poorly both increases the accumulation of positive rewards and decreases the frequency of incurred penalties. We found that, in all environments tested, the same risk profile can be used to produce both stronger overall performance than standard Proximal Policy Optimization (PPO) and higher levels of positive reward than PPO constrained by Lagrangians to maintain the same cost levels.
sections.heading: 1 INTRODUCTION
sections.text: While deep reinforcement learning (DRL) has been used to master an impressive array of simulated tasks in controlled settings, it has not yet been widely adopted for high-stakes, real-world applications. One reason for this gap is the lack of distributional perspective in standard artificial agents. Endowing agents with such perspective could make their decision-making more robust, potentially leading to increased safety, increased trust from humans, and more widespread real-world adoption.
  In reinforcement learning (RL), risk arises due to uncertainty around the possible outcomes of an agent’s future actions. It is a result of randomness in the operating environment, mismatch between training and test conditions, and the inherent randomness of a stochastic policy. Risk-sensitive policies, or those that consider more than a mean over the distribution of possible outcomes, offer the potential for added robustness under uncertain and dynamic conditions. There is an evolving landscape of algorithmic paradigms for handling risk in RL, from constraint-based approaches adapted from optimal control (Achiam et al., 2017; Chow et al., 2019; Ray et al., 2019; Zhong et al., 2020) to adversarial approaches emerging from AI Safety (Garcı́a & Fernández, 2015; Amodei et al., 2016). Within this landscape, learning approaches that optimize distributional measures offer the ability to express design preferences over the full distribution of potential outcomes, through the specification of a risk-sensitivity criterion.
  Distributional RL has been studied for value-based methods, with a popular strategy being to use the distributional Bellman equation to estimate the distribution of Q-values for each member of a discrete set of potential actions (Bellemare et al., 2017; Dabney et al., 2018a;b). However, distributional RL has not been widely explored for policy gradient methods, which could permit direct optimization of risk-sensitive measures and naturally accommodate both discrete and continuous action spaces.
  In the following, we introduce a novel framework for risk-sensitive learning using policy gradients. Our approach allows agents to be trained with different risk profiles through design-time specification of both utility and weight functions, with the latter being defined over the estimated distribution of full-episode rewards. This framework enables agent-based learning that captures aspects of human decision-making, such as overemphasis of rare occurrences and diminishing marginal utility relative to a reference outcome (Kahneman & Tversky, 1979). It also allows implementation of another key strategy of human learning: emphasizing improvement on tasks where one is deficient. We demonstrate the ability of our algorithm to use this strategy to improve performance relative to both unconstrained and constrained methods in six OpenAI Safety Gym environments (Ray et al., 2019).
sections.heading: 2 RELATED WORK
sections.text: Constrained RL offers a set of approaches to safe exploration (Garcı́a & Fernández (2015)) that aim to enforce explicit constraints throughout the learning process via methods including Lagrangian constraints (Ray et al. (2019)) and constraint coefficients (Achiam et al. (2017)). Differing from the safe exploration scenario, we here consider problems with distinct training and test phases where agent performance is to be evaluated. Our experiments indicate that risk-sensitive learning can offer performance improvements over constrained learning in scenarios where safety constraints need not be enforced during training.
  Distributional approaches to risk-sensitive RL have primarily been explored in the value-based setting. Therein, the value distribution has been explicitly modeled through categorical techniques (Bellemare et al., 2017) or quantile regression (Dabney et al., 2018a) and used to improve both value predictions and overall performance. Recent works utilize distributional modeling in the actor-critic setting to enable application to continuous action spaces, again demonstrating improved performance over baseline approaches (Ma et al., 2020; Zhang et al., 2021; Duan et al., 2021). In value-based approaches, risk-sensitivity criteria are applied at run time as a nonlinear warping of the estimated value distribution.
  Policy gradient approaches offer additional promise for risk-sensitive RL, but require direct optimization of a parameterized policy with respect to a distributional objective. Some existing methods are limited to a specific class of learning objective, such as the set of concave risk measures that allow a globally-optimal solution (Tamar et al., 2015; Zhong et al., 2020). Others allow a broader class of measures but are more restrictive in the class of policies that can be represented (Prashanth et al., 2016). We aim for a risk-sensitive policy gradient approach that both offers significant flexibility in the choice of learning objective and can learn policies parameterized by a deep neural network.
  Various measures have been considered in the context of risk-sensitive RL, including exponential utility (Pratt, 1964), percentile performance criteria (Wu & Lin, 1999), value-at-risk (Leavens, 1945), conditional value-at-risk (Rockafellar & Uryasev, 2000), and prospect theory (Kahneman & Tversky, 1979). In this work, we consider a class of risk-sensitivity measures motivated by Cumulative Prospect Theory (CPT) (Tversky & Kahneman, 1992). CPT uniquely models two key aspects of human decision-making: (1) a utility function u, computed relative to a reference point that induces more risk-averse behavior in the presence of gains than losses and (2) a weight function w that prioritizes outlying events. Specific forms of u and w are given in Tversky & Kahneman (1992) (and in Appendix A.4), but the general form of CPT admits a wide variety of risk-sensitive objectives.
  Here we show how to train agents to optimize this class of objectives through sampling-based estimation of their policy gradients and requisite variance reduction. The final algorithm resembles well-known on-policy approaches such as Proximal Policy Optimization (Schulman et al., 2017b) and is similarly widely applicable. Although we do not explore it here, the incorporation of an appropriate risk-sensitivity criterion could additionally enable risk-aware exploration and adversarial training for increased robustness (Pinto et al., 2017; Parisi et al., 2019; Zhang et al., 2020).
sections.heading: 3 RISK-SENSITIVE POLICY OPTIMIZATION
sections.text: In this section we formalize the class of distributional objectives to be considered, derive a samplingbased approximation of its policy gradient, enact variance reduction on this estimate, and use the result to produce a practical learning algorithm.
sections.heading: 3.1 PRELIMINARIES: PROBLEM AND NOTATION
sections.text: Standard deep reinforcement learning seeks to maximize the expected reward of an agent over encountered trajectories; that is, it maximizes the objective
  J(θ) = Eτ∼pθ(τ) [∑ t r(st,at) ] . (1)
  Here pθ(τ) is the distribution over trajectories τ ≡ s1,a1, . . . , sT ,aT induced by a policy parameterized by θ; st, at, and r(st,at) denote the state, action, and reward at time t, respectively. To enable the incorporation of distributional context, we instead consider the objective
  J(θ) = ∫ +∞ −∞ u(r(τ)) d dr(τ) ( w(Pθ(r(τ)) ) dr(τ), (2)
  where u(r(τ)) is the utility associated with full-trajectory reward r(τ) ≡ ∑ t r(st,at) and w
  is a piecewise differentiable weighting function of the CDF of trajectory reward Pθ(r(τ)) =∫ r(τ) −∞ pθ(r ′)dr′.
  Equation 2 is inspired by CPT (Tversky & Kahneman, 1992), which includes a pair of integrals of this form. It was chosen for its generality; by using different utility functions u and weight functions w one may represent all of the risk measures mentioned in Section 2 and all of the risk measures evaluated by Dabney et al. (2018a). The form (2) reduces to (1) when u and w are both the identity mapping. While designed for the episodic setting, the objective (2) may be considered for infinite horizons through the choice of appropriately long windows.
sections.heading: 3.2 RISK-SENSITIVE POLICY GRADIENT
sections.text: To optimize the objective (2), we first derive an approximation to its gradient with respect to the policy parameters θ. Working toward a representation that can be sampled, we assert the independence of the reward on θ and use the chain rule to write
  ∇θJ(θ) = ∫ ∞ −∞ u(r(τ)) d dr(τ) ( w′(Pθ(r(τ)))∇θPθ(r(τ)) ) dr(τ), (3)
  where w′ is the derivative of w with respect to Pθ(r(τ)). The gradient of the CDF may be written as follows:
  ∇θPθ(r(τ))=∇θ ∫ r(τ) −∞ pθ(r ′)dr′ = ∇θ ∫ τ ′ H(r(τ)− r(τ ′))pθ(τ ′)dτ ′
  = ∫ τ ′ H(r(τ)−r(τ ′))∇θpθ(τ ′)dτ ′ = ∫ τ ′ H(r(τ)−r(τ ′))pθ(τ ′)∇θ log pθ(τ ′)dτ ′. (4)
  Here the first equality is the integral representation of Pθ(r(τ)), the second uses the Heaviside step function to select all trajectories with total reward ≤ r(τ), the third follows from the independence of reward on θ, and the fourth follows from the expression for the derivative of the natural logarithm. In the following, we also use the complementary expression
  ∇θPθ(r(τ)) = ∇θ ( 1− ∫ ∞ r(τ) pθ(r ′)dr′ ) = − ∫ τ ′ H(r(τ ′)− r(τ))pθ(τ ′)∇θ log pθ(τ ′)dτ ′. (5)
  Either form, or a combination of the two, may be substituted into (3) and the result sampled over N trajectories by first ordering trajectories i = 1 . . . N by increasing reward r(τ). Then
  ∇θJ(θ) ≈ N∑ i=1 u(r(τi)) ( w′ ( i N ) ∇θPθ(r(τi))− w′ ( i− 1 N ) ∇θPθ(r(τi−1)) ) , (6)
  where the term w′(0)∇θPθ(r(τ0)) ≡ 0. This ordering scheme produces an asymptotically consistent estimate, as shown in the context of CPT value estimation by Prashanth et al. (2016). ∇θPθ(r(τi)) may be sampled in one of two ways, based on either (4) or (5):
  ∇θPθ(r(τi)) ≈ 1
  N i∑ j=1 Tj∑ t=1 ∇θ log πθ(aj,t|sj,t) ≈ − 1 N N∑ j=i+1 Tj∑ t=1 ∇θ log πθ(aj,t|sj,t). (7)
  The expression (6) may be used to train a policy that optimizes the distributional objective (2) in a manner similar to REINFORCE (Williams, 1992).
sections.heading: 3.3 VARIANCE REDUCTION
sections.text: Reducing the variance of sample-based gradient estimates enables faster learning. Here we take several steps to reduce the variance of (6), as has been done with the policy gradient estimate of REINFORCE (Williams, 1992).
  First, note that cross-trajectory terms of the form f(τi,aj,t, sj,t) = u(r(τi))∇θ log πθ(aj,t|sj,t), while nonzero, do not contribute to the gradient estimate in expectation when i 6= j. A proof of this assertion is given in Appendix A.1. Using (4) for the first term of (6) and (5) for the second allows us to write
  ∇θJ(θ) ≈ N∑ i=1 u(r(τi)) ( w′ ( i N ) 1 N i∑ j=1 Tj∑ t=1 ∇θ log πθ(aj,t|sj,t)
  +w′ ( i− 1 N ) 1 N N∑ j=i Tj∑ t=1 ∇θ log πθ(aj,t|sj,t) ) .
  (8)
  Removing cross-trajectory terms gives
  ∇θJ(θ) ≈ 1
  N N∑ i=1 u(r(τi)) (( w′ ( i N ) + w′ ( i− 1 N )) Ti∑ t=1 ∇θ log πθ(ai,t|si,t). (9)
  Note that the weight coefficients (w′( iN ) + w ′( i−1N )) should be normalized over each batch. The expression (9) is equivalent to (6) in expectation, but with reduced variance (see Appendix A.1 for justification). It has a clear intuition – trajectories are assigned utilities based on their rewards and their contributions to the gradient are scaled by the derivative of the weight function, just as they are in Cumulative Prospect Theory (Tversky & Kahneman, 1992).
  Standard variance reduction techniques may be applied to this simplified form. Without further assumption or introduction of additional bias, a static baseline b can be employed:
  ∇θJ(θ) ≈ 1
  N N∑ i=1 ( u(r(τi))− b )( w′ ( i N ) + w′ ( i− 1 N )) Ti∑ t=1 ∇θ log πθ(ai,t|si,t) (10)
  Justification for this assertion is given in Appendix A.2. Learning may be further improved if we additionally assume that utility may be allocated on a per-step basis. In this case, per-step utilities are computed as the difference between what the full-episode utility would be if the episode were to end at a given time step and what it would have been had the episode ended at the previous time step. While not applicable in cases where episode utility is adjusted based on final outcome, this assumption has the significant benefit of modeling the temporal allocation of rewards and aligns with the standard formulation of RL. With it, the variance of (9) may be further reduced through the incorporation of utility-to-go and a state-dependent baseline Vφ(si,t):
  ∇θJ(θ)≈ 1
  N N∑ i=1 [ w′ ( i N ) +w′ ( i− 1 N )] Ti∑ t=1 ∇θ log πθ(ai,t|si,t) [ Ti∑ t′=t u(si,t′ ,ai,t′)−Vφ(si,t) ] (11)
  Here u(si,t′ ,ai,t′) is the per-step utility. The value function Vφ(si,t) is parameterized by φ and trained via regression to minimize
  L(φ) = ∑ i,t ( Vφ(si,t)− Ti∑ t′=t u(si,t′ ,ai,t′) )2 . (12)
  A standard argument, similar to the approach taken in (Achiam, 2018), can be used to show that the incorporation of utility-to-go does not change the expected value of (9). The addition of a state-dependent baseline also does not introduce additional bias, as justified in Appendix A.2.
  Finally, discount factors, bootstrapping, and trust regions may be used to provide additional variance reduction, just as they are in conventional on-policy learning (Appendix A.2). These measures may introduce additional bias to the policy gradient estimate, but typically lead to more sampleefficient learning. In our experiments, we evaluate the use of generalized advantage estimation (GAE; (Schulman et al., 2016)) based on the utility-to-go as well as the clipping-based trust regions of Proximal Policy Optimization (PPO; (Schulman et al., 2017b)). Incorporating these in the policy gradient yields
  ∇θJ(θ) ≈ 1
  N N∑ i=1 ( w′ ( i N ) +w′ ( i− 1 N )) Ti∑ t=1 ∇θLclip ( log πθ(ai,t|si,t), Aπu(si,t,ai,t) ) , (13)
  where Aπu(si,t,ai,t) is the standard GAE except with per-step utilities in place of rewards. Trust regions are implemented similarly to PPO, pessimistically clipping policy updates to be within a multiplicative factor of 1± of the existing policy:
  Lclip = min ( log πθ(ai,t|si,t)Aπu(si,t,ai,t),
  log ( clip ( πθ(ai,t|si,t) πθold(ai,t|si,t) , 1± ) πθold(ai,t|si,t) ) Aπu(si,t,ai,t) ) .
  (14)
  They are used to perform multiple policy updates using the same batch of data, providing learning that is no longer strictly on-policy but that can be significantly more sample efficient. When following this route, we apply the same early stopping mechanism, based on the Kullback-Leibler divergence (DKL) between old and new policies, as was used by Ray et al. (2019).
sections.heading: 3.4 LEARNING ALGORITHM
sections.text: The above sample-based estimate of the policy gradient may be used to train agents to maximize distributional objectives of the form (2). The resulting method, Cumulative Prospect Proximal Policy Optimization (C3PO), is given in Algorithm 1 and mirrors standard on-policy learning.
  Algorithm 1 Cumulative Prospect Proximal Policy Optimization (C3PO) Require: Policy: initial parameters θ0, learning rate αθ, updates per batch Mθ Require: Value: initial parameters φ0, learning rate αφ, updates per batch Mφ Require: Early stopping threshold DKL, stop, discount factor γ
  for k = 1, 2, . . . do Collect set of episodes Dk = {τi} by running policy π(θk) in the environment Compute per-step utilities u(si,t,ai,t) Fit value function by regression: for m = 1, . . .Mφ do
  φ← φ+ αφ∇φ 1∑ i Ti ∑ i,t ( Vφ(si,t)− ∑Ti t′=t γ t′−tu(si,t′ ,ai,t′) )2 end for Update utility-based advantage estimates Aπu(s,a), using new Vφ(s) Compute weight coefficients based on ordered episode outcomes and normalize Update policy, using KL-based early stopping: for m = 1, . . .Mθ do
  if DKL(πθ||πθold) < DKL, stop then θ ← θ + αθ 1N ∑N i=1(w ′( iN ) + w ′( i−1N )) ∑Ti t=1∇θLclip(log πθ(ai,t|si,t), Aπu(si,t,ai,t)) else break
  end if end for
  end for
  Beyond the utility and weight components, Algorithm 1 differs from conventional methods in the requirement to collect full episodes of data in each batch. This requirement can be removed if outcomes can be defined over partial rather than full episodes, an assumption that is often viable and matches human decision-making. For instance, while out of scope for this work, our approach could be applied to the Atari suite (Bellemare et al., 2013) by considering the outcomes of fixed-length windows and restructuring Algorithm 1 to mimic minibatch PPO (Schulman et al., 2017b).
sections.heading: 4 EXPERIMENTS
sections.text: To evaluate our approach, we sought to both establish that it can effectively optimize different distributional objectives and explore the impact of using different objectives on agent outcomes. We found the OpenAI Safety Gym (Ray et al., 2019) to be suitable for these purposes. Safety Gym is a configurable suite of continuous, multidimensional control tasks wherein different types of robots must navigate through obstacles with different dynamics to perform different tasks. By including both positive and negative events in each training scenario, it allowed us to evaluate how our various agents handled risk. Safety Gym is also highly stochastic: the locations of the goals and obstacles are randomized, leading to outcome variability and forcing the agent to learn a generalized navigation strategy.
  Safety Gym logs adverse events but does not incorporate them into the reward function. As our method relies solely on the training signal from the reward, we assigned each logged adverse event a fixed, negative reward contribution in experiments using it or other unconstrained agents. Our initial experiments were conducted with a reward contribution of −0.025, which was found to allow agents to prioritize reaching goals but deter them from collisions with obstacles. To further emphasize obstacle avoidance, we doubled this contribution to −0.05 in our experiments using cautious weightings. These choices and the role they play are further discussed in Section 5.
  To highlight distributional differences, we focused on the publicly available, obstacle-rich level 2 environments.1 Avoiding the longer compute time of the “Doggo” robot, we evaluated the “Point” and “Car” robots on each task (“Goal”, “Button”, and “Push”). Further details on these environments and our rationale for choosing them are given in Appendix A.3.
  In all experiments, we evaluated five random seeds and matched the hyperparameters used in the baselines accompanying Safety Gym (Ray et al., 2019) as closely as possible. The neural networks used to model both policy and value were multilayer perceptrons (MLPs), with two hidden layers of 256 units each and tanh activations. As in Ray et al. (2019), the policy network outputs the mean values of a multivariate gaussian with diagonal covariance. The control variances are optimized but independent of state. The full complement of variance reduction measures were used throughout; see Appendix A.4 for experimental justification of this choice.
sections.heading: 4.1 DIFFERING OBJECTIVES
sections.text: Agent performance was explored under four different distributional objectives. In addition to expected reward and CPT (configured to match the original form of Tversky & Kahneman (1992) and as given in Appendix A.4), we optimized for cautious (η = 0.75) and aggressive (η = −0.75) versions of the
  1In Safety Gym, the default environments have three levels (0, 1, 2); obstacle density increases with level.
  .
  distortion risk measure proposed in Wang (2000). This measure is defined as w(p) = Φ(Φ−1(p)+η), where Φ and Φ−1 are the standard normal cumulative distribution function and its inverse. While we found this form to be convenient, the “Pow” metric in Dabney et al. (2018a) or any other set of similarly shaped w curves should achieve a similar effect. In experiments using the objective from Tversky & Kahneman (1992), the reference point was taken to be the mean episode reward of the current batch, matching the tendency of humans to change their standards over time. The four weight functions and their resulting coefficients in (9) are shown in Figure 1.
  Plots of the total rewards (including penalties) in training, average cost events per episode in training, and outcome distributions in testing are shown for one environment in Figure 2 and for two additional environments in Appendix A.5. The trends were fairly consistent over the three environments evaluated in this manner. While no explicit effort was made to handle cost (agents were given only the sum of positive rewards and penalties), the cautious and aggressive weightings consistently accumulated relatively low and high costs, respectively. The cautious (Wang(η = 0.75)) agent typically also generated the highest positive and total rewards after 10 millions steps of training.
  To generate the histograms in Figures 2, 5, 8, 9, and 13 as well as the numbers in Table 1, the trained agents were deployed on a set of 5000 test episodes – 1000 for each of the 5 networks learned using different random seeds in training. The resulting distributions therefore include contributions from both aleatoric and epistemic uncertainty. Sampling was turned off, allowing the agents to choose their perceived optimal action at each time step. In this context the benefit of emphasizing the lower part of the outcome distribution (i.e., cautious weighting) became more pronounced, in part because the methods that emphasize poor outcomes tended to maintain higher policy entropy (Appendix A.5).
sections.heading: 4.2 CAUTIOUS WEIGHTINGS
sections.text: To further explore the apparent benefits of cautious weightings in Safety Gym, we trained a series of variably cautious agents by tuning η in the risk-averse weight function proposed by Wang (2000). Histograms of their episode rewards in testing are given in Appendix A.5 and summarized in Table 1. In these environments, agent performance – both in terms of improving the lower end of the reward distribution and on average – was seen to generally improve with increasing η until the range η ∈ [0.75, 1.25], subsequently degrading. Additional comparisons were made with PPO (Schulman et al., 2017b), which unsurprisingly was found to closely track performance of the “Uniform” agent. We found that naively incorporating cautious weightings into PPO improved its performance (row PPO + Wang(0.75) in Table 1), though not to the level of the full C3PO method with η = 0.75.
  We then pursued a set of longer runs to compare C3PO with the cautious objective from Wang (2000) to both unconstrained and constrained benchmarks. Here we did not tune η, keeping it fixed at 0.75 for all experiments. Comparisons with unconstrained methods for three environments are given in Figures 3, 4, and 5 and for the remaining three environments in Appendix A.6. In addition to PPO, we compared performance with Trust Region Policy Optimization (TRPO; Schulman et al. (2017a)) as configured in Ray et al. (2019). Since this TRPO configuration generally outperformed the PPO configuration in Ray et al. (2019) from which we derived the hyperparameters for C3PO, we would expect C3PO to be at a disadvantage compared to TRPO. However, we found C3PO had the highest
  average reward (including penalty) in five of the six environments and lowest average cost in five of the six environments. In addition, agents that used the cautious weightings tended to have more stable and repeatable training, as evidenced by the tight distribution of their learning curves. This tightness was found to reflect a lack of negative outlier episodes and potentially lower epistemic uncertainty throughout training. Finally, note that the use of a nonzero penalty for cost events resulted in significantly lower incurred costs than were observed with unconstrained agents trained without a penalty (Ray et al., 2019). PPO and TRPO were seen to reach similar cost levels without a penalty; these levels are indicated by red dashed lines in the cost figures.
  Comparisons with versions of PPO and TRPO that use Lagrangian constraints (PPO-Lagrangian and TRPO-Lagrangian; Ray et al. (2019)) to match the cost level of C3PO are shown in Figure 6 and Appendix A.7 . We see that, given the same level of cost incurred per episode, agents trained using C3PO consistently achieve higher levels of reward than those trained with PPO-Lagrangian and TRPO-Lagrangian. As above, training is seen to be more stable and repeatable using our risksensitive method. Additional comparisons were generated with Constrained Policy Optimization (CPO; Achiam et al. (2017)), but are not shown in Figure 6 because they failed to maintain the cost levels of the other methods. For completeness, they are given in Appendix A.7.
sections.heading: 5 DISCUSSION
sections.text: The analysis above allows for sample-based policy gradient estimates of a broad class of distributional objectives. Variance reduction measures were shown to enable efficient optimization based on these estimates (Appendix A.4). However, it was not seen to be the case that a given distributional objective could be most effectively optimized directly. Instead, the best results were generally obtained through moderate emphasis on improving negative training outcomes (Table 1).
  To understand this behavior, consider the interplay of optimization and exploration in the training of cautious and aggressive agents. Cautious weightings continually emphasize the lower part of the outcome distribution, pushing that part of the distribution upward and adjusting behavior the most where it is most necessary. Once a part of the state space where the agent is deficient is adequately addressed, a different part of the state space takes its place. Policy entropy remains high because of the emphasis on problematic situations, ensuring adequate exploration. This trend continues with increasing η, until the point where the agent begins to ignore high quality training outcomes too much. Conversely, aggressive weightings continually emphasize the best outcomes in the distribution. When an already strong outcome is given increased attention, it is likely to stay at the top. Hence agents trained with aggressive weightings tend to become myopic, obsessing over a fraction of the state space while neglecting the rest of it. They tend to explore inadequately and ironically fail to attain better top-end performance than more cautious weightings.
  Given the consistent performance gains observed using our method, we propose that it represents a useful option for improving the performance and stability of on-policy learners. This should be particularly true in the presence of a meaningful trade-off between positive and negative reward terms and when there is significant stochasticity. While our approach does add an additional hyperparameter – the shaping constant η – one value for that hyperparameter was seen to provide gains across all environments tested. While our approach does not provide for a direct choice of cost limit as constrained methods do, it is simpler to implement and was consistently seen to be more performant for the cost level it reached. It is also likely possible to use cautious weightings in conjunction with constrained RL, though this has not yet been investigated.
sections.heading: 6 CONCLUSIONS
sections.text: In this work, we proposed a risk-sensitive learning algorithm based on a policy gradient estimate for a broad class of distributional objectives. When configured to emphasize improvement in scenarios where the agent performs poorly, we found our method to compare favorably with existing unconstrained and constrained on-policy learners.
sections.heading: A APPENDIX
sections.heading: A.1 EVALUATION OF CROSS-TRAJECTORY TERMS IN POLICY GRADIENT
sections.text: In this section, we first show that the cross-trajectory terms in our policy gradient estimate (6) have an expectation value of 0. We then argue that their removal leads to a policy gradient estimate with reduced variance.
  Lemma 1. Cross-trajectory terms of the form f(τi,aj,t, sj,t) = u(r(τi))∇θ log πθ(aj,t|sj,t), where i 6= j, do not contribute to the gradient estimate (6) in expectation.
  Proof. First, note that
  Eτi∼pθ(τ),τj∼pθ(τ)f(τi,aj,t, sj,t) = Eτi∼pθ(τ),τj∼pθ(τ)u(r(τi))∇θ log πθ(aj,t|sj,t)
  = Eτi∼pθ(τ) [ u(r(τi))Eτj∼pθ(τ) [ ∇θ log πθ(aj,t|sj,t) ∣∣∣∣τi] ]
  Then consider the innermost expectation:
  Eτj∼pθ(τ) [ ∇θ log πθ(aj,t|sj,t) ∣∣∣∣τi] = ∫ sj,t,aj,t p(sj,t,aj,t|πθ, τi)∇θ log πθ(aj,t|sj,t)daj,tdsj,t
  = ∫ sj,t p(sj,t|πθ, τi) ∫ aj,t πθ(aj,t|sj,t)∇θ log πθ(aj,t|sj,t)daj,tdsj,t
  = ∫ sj,t p(sj,t|πθ, τi) ∫ aj,t ∇θπθ(aj,t|sj,t)daj,tdsj,t
  = ∫ sj,t p(sj,t|πθ, τi)∇θ ∫ aj,t πθ(aj,t|sj,t)daj,tdsj,t
  = ∫ sj,t p(sj,t|πθ, τi)(∇θ1)dsj,t = 0.
  To see why removal of cross-trajectory terms leads to reduced variance, consider that the full expression (6) may be written as the sum of terms of the form f(τi,aj,t, sj,t) = u(r(τi))∇θ log πθ(aj,t, sj,t). Its variance is the sum of the total variance from terms where i = j, the total variance from terms where i 6= j, and a term proportional to the covariance of these two totals. However, because each term in the covariance contains at least one trajectory that differs from the rest, the above reasoning may be applied to argue that the covariance is 0. Hence, the removal of the cross-trajectory terms lowers the variance of the policy gradient estimate by the variance of the cross-trajectory terms.
  A.2 INTRODUCTION OF STATIC AND STATE-DEPENDENT BASELINES
  Lemma 2. A static baseline of the utility may be added to the policy gradient estimate (9) without introduction of bias.
  Proof. The additional term is 0 in expectation as
  Eτi∼pθ(τ)
  [ b ( w′ ( i
  n
  ) + w′ ( i− 1 n )) ∇θ log πθ(ai,t|si,t) ] =b ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t,ai,t p(si,t,ai,t|πθ) [ ∇θ log πθ(ai,t|si,t) ] dai,tdsi,t
  =b ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t p(si,t|πθ) ∫ ai,t πθ(ai,t|si,t)∇θ log πθ(ai,t|si,t)dai,tdsi,t
  =b ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t p(si,t|πθ)∇θ ∫ ai,t πθ(ai,t|si,t)dai,tdsi,t
  =b ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t p(si,t|πθ)(∇θ1)dsi,t = 0.
  The contribution of the weight terms (w′( in ) + w ′( i−1n )) may be pulled out of the integral between the first and second line because of its independence on both state and action. This term is fixed for a given trajectory by the rank of its reward amongst the rewards accumulated on all trajectories in the current batch.
  In our variance reduction experiments (Appendix A.4), the “Base” agent uses b equal to the mean of full-episode utility in the current batch.
  As described in Section 3.3, we may further adjust the policy gradient estimate through introduction of per-step utilities. In this case, we may justify the use of a state-dependent baseline through the following.
  Lemma 3. A state-dependent baseline Vφ(si,t) may be added to the policy gradient estimate (9) without introduction of bias, if per-step utilities are assumed.
  Proof. The additional term is 0 in expectation as
  Eτi∼pθ(τ)
  [( w′ ( i
  n
  ) + w′ ( i− 1 n )) ∇θ log πθ(ai,t|si,t)Vφ(si,t) ] = ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t,ai,t p(si,t,ai,t|πθ)Vφ(si,t)∇θ log πθ(ai,t|si,t)dai,tdsi,t
  = ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t p(si,t|πθ)Vφ(si,t) ∫ ai,t πθ(ai,t|si,t)∇θ log πθ(ai,t|si,t)dai,tdsi,t
  = ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t p(si,t|πθ)Vφ(si,t)∇θ ∫ ai,t πθ(ai,t|si,t)dai,tdsi,t
  = ( w′ ( i
  n
  ) + w′ ( i− 1 n ))∫ si,t p(si,t|πθ)Vφ(si,t)(∇θ1)dsi,t = 0.
  The rationale for pulling the w′ terms out of the integral is the same as in Lemma 2.
  Finally, we note that the ability to pull the contribution of the weight terms (w′( in ) +w ′( i−1n )) to the front of Equation 11 allows us to formulate advantage estimates based on per-step utility. Bootstrap estimates of the value function Vφ(si,t) and Generalized Advantage Estimation as in Schulman et al. (2016) can be conducted exactly as they are in standard on-policy learning, if rewards are replaced by per-step utilities.
sections.heading: A.3 ADDITIONAL INFORMATION ON SAFETY GYM
sections.text: As mentioned in Section 4, we chose to evaluate our approach using the OpenAI Safety Gym (Ray et al., 2019). The choice was governed by our desire to test in conditions with clear cost-benefit trade-offs, significant stochasticity, adequate complexity, and available benchmarks. While our methods are not limited to particular task types or observation/action spaces, we found Safety Gym to be suitable for exploring their potential.
  The six environments chosen were the most obstacle-rich of the publicly available environments that used the “Point” and “Car” robots. The Point robot is constrained to the 2D plane and has two control dimensions: one for moving forward/backward and one for turning. The Car robot also has two control dimensions, corresponding to independently actuated parallel wheels. It has a freely rotating wheel and, while it is not constrained to the 2D plane, typically remains in it. While we expect our results to extend to the remaining default robot, “Doggo”, we did not experiment with it because of the order of magnitude longer training times it exhibited in Ray et al. (2019).
  Several types of obstacles and tasks were present in the environments we evaluated. In all cases, the robot is given a fixed amount of time (1000 steps) to complete the prescribed task as many times as possible and is motivated by both sparse and dense reward contributions. In the “Goal” environments, the robot must navigate to a series of randomly-assigned goal positions, with a new target being assigned as soon as a goal is reached. In the “Button” environments, the robot must reach and press a sequence of goal buttons while avoiding other buttons. In the “Push” task, the robot must push a box to a series of goal positions. The set of obstacles are different for each task; among the three environments there are a total of five different constraint elements (hazards, vases, incorrect buttons, pillars, and gremlins), each with different dynamics. See Ray et al. (2019) for further details.
sections.heading: A.4 EMPIRICAL PERFORMANCE OF VARIANCE REDUCTION MEASURES
sections.text: To gauge the impact of the variance reduction techniques outlined in Section 3.3, we evaluated their performance in maximizing the value function of Cumulative Prospect Theory (Tversky & Kahneman, 1992). As mentioned in Section 3.1, this function has two integrals of the form (2):
  J(θ) = − ∫ ∞ −∞ u−(r(τ)) d dr(τ) ( w−(Pθ(r(τ))) ) dr(τ)
  + ∫ ∞ −∞ u+(r(τ)) d dr(τ) ( − w+(1− Pθ(r(τ))) ) dr(τ)
  (15)
  In Tversky & Kahneman (1992), the utility functions are computed relative to a reference point and reflect the tendency of humans to be more risk-averse in the presence of gains than in the presence of losses. The weight functions {w+, w−} model our inclination to emphasize the best and worst possible outcomes in our decision-making.
  More specifically, in these experiments we used the piecewise utility functions u+(r) = H(r − r0)(r− r0)σ and u− = λH(r0− r)(r0− r)σ with static reference r0 = 10, σ = 0.88, and λ = 2.25. The weight function w(p) = p η
  (pη+(1−p)η) 1 η
  was used, where η = 0.61 for r < r0 and η = 0.69 for
  r ≥ r0. Four methods were evaluated, incorporating progressive amounts of variance reduction:
  • Base: Risk-sensitive policy gradient with a static baseline (10)
  • UTG: Base with utility-to-go and a neural network baseline (11)
  • GAE: UTG with generalized advantage estimation ((13) without clipping)
  • TR: GAE with trust regions ((13) with clipping (14))
  As shown in Figure 7, the incorporation of these techniques increased the sample efficiency of the CPT value optimization significantly. Consequently, we used the full complement (TR) in all other experiments.
sections.heading: A.5 DIFFERING OBJECTIVES: ADDITIONAL RESULTS
sections.text: Below we include results for all environments for the experiments described in Section 4.1.
  In addition, we note the trend of policy entropies with different distributional objectives. In general, more cautious weightings maintain higher entropy for longer than more aggressive weightings. Note that these plots represent an upper bound because they do not account for action clipping by the environment; see Ray et al. (2019) for details.
sections.heading: A.6 ADDITIONAL COMPARISONS WITH UNCONSTRAINED METHODS
sections.text: Below are plots of average episode reward and average number of episode cost events throughout training for the remainder of the environments on which we conducted long runs (Section 4.2). Also included are histograms of testing performance for those runs.
sections.heading: A.7 ADDITIONAL COMPARISONS WITH CONSTRAINED METHODS
sections.text: As mentioned in Section 4.2, we compared the performance of C3PO with constrained methods by setting the cost limit of the constrained methods to match the cost level attained by C3PO. Here we provide
  • the positive reward plots for the remaining three Safety Gym environments studied, • the cost plots for each of the six environments, and
  • all plots for Constrained Policy Optimization (CPO; Achiam et al. (2017)).
  The intent of the cost plots of Figure 15 is to show rough consistency between the cost levels of our approach and Lagrangian methods configured to have the same cost limit. This is verified, but other trends should be noted. First, while the Lagrangian-based methods typically follow the cost constraint well, they cannot satisfy it in each batch. Second, our approach tends to have comparable or lower cost rates throughout training. This safe exploration metric, defined in Ray et al. (2019), refers to the average cost per episode over all of training up to a given point.
  Results related to Constrained Policy Optimization (CPO; Achiam et al. (2017)) are included here but not in the main text because, consistent with (Ray et al., 2019), we were not able to configure CPO to respect the cost levels of the other constrained methods. Here we show the cost levels reached by CPO compared with C3PO (Figure 16) as well as a comparison of the average episode rewards of the two (Figure 17). For the latter, we employed the penalty scaling used by C3PO to enable a fair comparison.
sections.heading: A.8 SUPPLEMENTARY MATERIALS
sections.text: The code used to produce these results is included in our Supplementary Materials.
title: A RISK-SENSITIVE POLICY GRADIENT METHOD