summaries: The paper proposes an Expected Quadratic Utility Maximization (EQUM) framework for policy grandient Mean-Variance control. The authors claim that current state-of-the-art methods suffer either suffer from computational issues or cannot control risk at desiderable level, hence they propose their approach as a possible solution. They provide different interpretation for the EQUM: standard objective with a regularization, variance minimization with a constraint on expected return and a return targeting optimization. A policy gradient algorithm for EQUM is proposed together with its Actor-Critic extension. The paper proposes a policy gradient style RL algorithm that optimizes an expected quadratic utility, a commonly used objective of risk management in finance and economics. The key idea here is based on the observation that when using the quadratic utility function, the use of mean-variance RL methods can be shown to optimize the utility of the agent. To this effect, the paper considers the use of expected quadratic utility maximization in the policy gradient. The quadratic utility can be naturally modeled using mean and variance. The paper implements two variations -- policy gradient and actor-critic with EQUM framework.  In this paper the author proposed a new mean-variance algorithm whose policy gradient algorithm is more simpler than other SOTA methods and it has an unbiased gradient. Instead of formulating the problem as an traditional mean variance constrained problem, the authors utilized quadratic utility theory and formulate the problem as variance minimization problem with a mean reward equality constraint. Then by reformulating the problem with the penalized problem and opening up the variance formulation, they showed that this mean-variance formulation does indeed have an unbiased policy gradient, that does not require advanced techniques such as double sampling or frenchel duality. To demonstrate the effectiveness of this method on balancing risk and return, they also evaluate their methods on several risk-sensitive RL benchmarks (such as portfolio optimization) and compared with a wide range of risk-sensitive RL methods.
abstractText: In real-world decision-making problems, risk management is critical. Among various risk management approaches, the mean-variance criterion is one of the most widely used in practice. In this paper, we suggest expected quadratic utility maximization (EQUM) as a new framework for policy gradient style reinforcement learning (RL) algorithms with mean-variance control. The quadratic utility function is a common objective of risk management in finance and economics. The proposed EQUM has several interpretations, such as reward-constrained variance minimization and regularization, as well as agent utility maximization. In addition, the computation of the EQUM is easier than that of existing mean-variance RL methods, which require double sampling. In experiments, we demonstrate the effectiveness of the EQUM in benchmark setting of RL and financial data.
references.authors: Jonathan Baxter Peter L. Bartlett
references.title: Infinite-horizon policy-gradient estimation
references.venue: Journal of Artificial Intelligence Research,
references.authors: J.O. Berger
references.title: Statistical Decision Theory and Bayesian Analysis
references.authors: Lorenzo Bisi Luca Sabbioni Edoardo Vittori Matteo Papini Marcello Restelli
references.title: Risk-averse trust region optimization for reward-volatility reduction
references.venue: In IJCAI,
references.authors: Taras Bodnar Nestor Parolya Wolfgang Schmid
references.title: On the Exact Solution of the Multi-Period Portfolio Choice Problem for an Exponential Utility under Return Predictability
references.venue: European Journal of Operational Research,
references.authors: Taras Bodnar Yarema Okhrin Valdemar Vitlinskyy Taras Zabolotskyy
references.title: Determination and estimation of risk aversion coefficients
references.venue: Computational Management Science,
references.authors: Stephen Boyd Lieven Vandenberghe
references.title: Convex Optimization
references.authors: Michael W Brandt
references.title: Portfolio choice problems
references.venue: In Handbook of financial econometrics: Tools and techniques
references.authors: T. Bulmuş S. Özekici
references.title: Portfolio selection with hyperexponential utility functions
references.venue: OR Spectrum,
references.authors: Yinlam Chow Mohammad Ghavamzadeh
references.title: Algorithms for cvar optimization in mdps
references.venue: In NeurIPS
references.authors: Yinlam Chow Mohammad Ghavamzadeh Lucas Janson Marco Pavone
references.title: Risk-constrained reinforcement learning with percentile risk criteria
references.venue: Journal of Machine Learning Research,
references.authors: Victor DeMiguel Lorenzo Garlappi Raman Uppal
references.title: Optimal versus naive diversification: How inefficient is the 1/n portfolio strategy
references.venue: The review of Financial studies,
references.authors: Yue Deng Feng Bao Youyong Kong Zhiquan Ren Qionghai Dai
references.title: Deep direct reinforcement learning for financial signal representation and trading
references.venue: IEEE Transactions on Neural Networks and Learning Systems,
references.authors: Eugene F Fama Kenneth R French
references.title: The cross-section of expected stock returns
references.venue: Journal of Finance,
references.authors: Peter C. Fishburn R. Burr Porter
references.title: Optimal portfolios with one safe and one risky asset: Effects of changes in rate of return and risk
references.venue: Management Science,
references.authors: Javier García Fernando Fernández
references.title: A comprehensive survey on safe reinforcement learning
references.venue: Journal of Machine Learning Research,
references.authors: Peter Geibel Fritz Wysotzki
references.title: Risk-sensitive reinforcement learning applied to control under constraints
references.venue: Journal of Artificial Intelligence Research,
references.authors: J.G. Kallberg W.T. Ziemba
references.title: Comparison of alternative utility functions in portfolio selection problems
references.authors: Yoram Kroll Haim Levy Harry Markowitz
references.title: Mean-variance versus direct utility maximization
references.venue: Journal of Finance,
references.authors: John Lintner
references.title: The valuation of risk assets and the selection of risky investments in stock portfolios and capital budgets
references.venue: The Review of Economics and Statistics,
references.authors: H. Markowitz
references.title: Portfolio Selection: Efficient Diversification of Investments
references.authors: Harry Markowitz
references.title: Portfolio selection
references.venue: The journal of finance,
references.authors: H.M. Markowitz G.P. Todd W.F. Sharpe
references.title: Mean-Variance Analysis in Portfolio Choice and Capital Markets
references.authors: Robert Merton
references.title: Lifetime portfolio selection under uncertainty: The continuous-time case
references.venue: The Review of Economics and Statistics,
references.authors: Volodymyr Mnih Adria Puigdomenech Badia Mehdi Mirza Alex Graves Timothy Lillicrap Tim Harley David Silver Koray Kavukcuoglu
references.title: Asynchronous methods for deep reinforcement learning
references.authors: Jan Mossin
references.title: Equilibrium in a capital asset
references.venue: market. Econometrica,
references.authors: Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury Gregory Chanan Trevor Killeen Zeming Lin Natalia Gimelshein Luca Antiga Alban Desmaison Andreas Kopf Edward Yang Zachary DeVito Martin Raison Alykhan Tejani Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala
references.title: Pytorch: An imperative style, high-performance deep learning library
references.venue: In NeurIPS,
references.authors: L.A. Prashanth Mohammad Ghavamzadeh
references.title: Actor-critic algorithms for risk-sensitive mdps
references.venue: In NeurIPS
references.authors: L.A. Prashanth Mohammad Ghavamzadeh
references.title: Variance-constrained actor-critic algorithms for discounted and average reward mdps
references.venue: Machine Learning,
references.authors: Martin L. Puterman
references.title: Markov Decision Processes: Discrete Stochastic Dynamic Programming
references.authors: John Schulman Sergey Levine Pieter Abbeel Michael Jordan Philipp Moritz
references.title: Trust region policy optimization
references.venue: In ICML,
references.authors: William F. Sharpe
references.title: Capital asset prices: A theory of market equilibrium under conditions of risk
references.venue: The Journal of Finance,
references.authors: Weiwei Shen Jun Wang
references.title: Portfolio blending via thompson sampling
references.venue: In IJCAI,
references.authors: Weiwei Shen Bin Wang Jian Pu Jun Wang
references.title: The kelly growth optimal portfolio with ensemble learning
references.venue: In AAAI,
references.authors: Richard S. Sutton Andrew G. Barto
references.title: Reinforcement Learning: An Introduction
references.authors: Richard S. Sutton David McAllester Satinder Singh Yishay Mansour
references.title: Policy gradient methods for reinforcement learning with function approximation
references.venue: NeurIPS,
references.authors: Aviv Tamar Dotan Di Castro Shie Mannor
references.title: Policy gradients with variance related risk criteria
references.venue: In ICML,
references.authors: Aviv Tamar Shie Mannor Huan Xu
references.title: Scaling up robust mdps using function approximation
references.venue: In ICML,
references.authors: J. Tobin
references.title: Liquidity Preference as Behavior Towards Risk
references.venue: Review of Economic Studies,
references.authors: R.J. Williams
references.title: Toward a Theory of Reinforcement-learning
references.venue: Connectionist Systems
references.authors: Ronald Williams Jing Peng
references.title: Function optimization using connectionist reinforcement learning algorithms
references.venue: Connection Science,
references.authors: Ronald J. Williams
references.title: Simple statistical gradient-following algorithms for connectionist reinforcement learning
references.venue: Machine Learning,
references.authors: Tengyang Xie Bo Liu Yangyang Xu Mohammad Ghavamzadeh Yinlam Chow Daoming Lyu Daesub Yoon
references.title: A block coordinate ascent algorithm for mean-variance optimization
references.venue: NeurIPS
references.authors: Shangtong Zhang Bo Liu Shimon Whiteson
references.title: Mean-variance policy iteration for risk-averse reinforcement learning, 2020
references.authors: W.T. Ziemba C. Parkan R. Brooks-Hill
references.title: Calculation of investment portfolios with risk free borrowing and lending
references.venue: Management Science,
references.authors: Xie
references.title: A ECONOMIC THEORY AND QUADRATIC UTILITY FUNCTION Considering the mean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al
references.authors: Xie
references.title: difficult; therefore, first gathering information based on analyses of an economist, then we construct the portfolio using the information and provide the set of the portfolios to an economic agent. However, owing to the recent development of machine learning, we can directly represent the complicated economic dynamics using flexible models, such as deep neural networks
references.authors: Ziemba
references.title: investigated the change of the portfolio proportion when the parameter of the quadratic utility function changes using the Canadian financial dataset. Recently, Bodnar et al. (2018) investigate the risk parameter (α and β in our formulation of the quadratic utility function) using the markets indexes in the world. They found that the utility function parameter depends on the market data model
references.authors: Wcall Wput respectively
references.title: The maturity time is set as τ =. If the buyer executes the option at time t, the buyer obtains a reward rt = max(0,Wput − xt)+max(0, xt−Wcall), where xt is an asset price. We set x0 = 1 and define the stochastic process as follows: xt = xt−1fu with probability 1 and xt = xt−1fd with probability 1, where fu and fd
references.authors: Tamar
references.title: We compare our methods with the standard SPG, and MVLR methods
references.authors: Tamar
references.title: SPG-based methods proposed by Tamar et al
references.venue: We choose the parameter of MVP from 1 and 10,
sections.heading: 1 INTRODUCTION
sections.text: Reinforcement learning (RL) with Markov decision processes (MDPs) is one type of dynamic decision-making problem (Puterman, 1994; Sutton & Barto, 1998). While the typical objective is the expected cumulative reward maximization, risk-aware decision-making has attracted great attention in real-world applications, such as finance and robotics (Geibel & Wysotzki, 2005; García & Fernández, 2015). The notion of risk is related to the fact that even an optimal policy may perform poorly owing to the stochastic nature of the problem. To capture the risk, various criteria have been proposed, such as Value at Risk (Luenberger, 1998; Chow & Ghavamzadeh, 2014; Chow et al., 2017) and variance (Markowitz, 1952; Markowitz et al., 2000; Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013). Among them, we focus on the mean-variance trade-off.
  Typical mean-variance RL (MVRL) methods attempt to maximize the expected cumulative reward while maintaining the variance threshold (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013; 2016; Xie et al., 2018; Bisi et al., 2020; Zhang et al., 2020). However, most existing MVRL methods suffer from high computational costs owing to the double sampling issue when approximating the gradient of the variance term (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013; 2016). To avoid the double sampling issue, Xie et al. (2018) proposed a method based on the Legendre-Fenchel duality (Boyd & Vandenberghe, 2004). Although the method does not suffer from the double sampling issue, we cannot apply a standard policy gradient method and must use a coordinate descent algorithm. In addition, the method cannot control risk at a certain desirable level.
  From an economics perspective, the difference in the RL objectives arises from the forms of utility functions. When the objective of an agent is expected cumulative reward maximization, the utility function is risk-neutral; when an agent attempts to control the risk based on an expected reward, the utility function is risk-averse (Mas-Colell et al., 1995). In economics, there have been several risk-averse utility functions proposed. The quadratic utility function is one such functions and is frequently used in financial economics (Luenberger, 1998). Under the quadratic utility function, the mean-variance portfolio maximizes the utility of the investor. In addition, other various financial theories are also based on the quadratic utility maximization (Markowitz, 1952; Sharpe, 1964; Lintner, 1965; Mossin, 1966). For more details, see Appendix A. In this study, as one of the MVRL approaches, we consider the expected quadratic utility maximization (EQUM) based on the policy gradient method (Williams, 1988; 1992; Sutton et al., 1999; Baxter & Bartlett, 2001).
  The EQUM has the following advantages: (i) low computational cost; (ii) numerous interpretations, and (iii) direct connections to real-world applications. In this study, as interpretations of EQUM, we propose the minimization of the variance under the constraint of the expected cumulative reward, reward-targeting optimization, and regularization. Thus, this study contributes to the context of riskaverse RL and MVRL. In the following sections, we first formulate the problem setting in Section 2 and review the existing methods in Section 3. Then, we propose the main algorithms in Section 4. Finally, we investigate the empirical effectiveness of the EQUM in Section 5.
sections.heading: 2 PROBLEM SETTING
sections.text: We consider the standard RL framework, where a learning agent interacts with an unfamiliar, dynamic, and stochastic environment modeled by a Markov decision process (MDP) in discrete time. We define the MDP as the tuple (S,A, r, P, P0), where S is a set of states, A is a set of actions, r : S × A → R is a reward function, P : S × S × A → [0, 1] is the transition kernel, and P0 : S → [0, 1] is an initial state distribution. The initial state S1 is sampled from P0. Let πθ : A × S → [0, 1] be a parameterized stochastic policy mapping states to actions, where θ is the tunable parameter. At time step t, an agent chooses an actionAt according to a policy πθ(· | St). We assume that the policy πθ is a differentiable function with respect to θ; that is, ∂πθ(a,s) ∂θ exists.
  There are several performance measures for a policy πθ. One popular measure is the expected cumulative reward from time step t to u defined as Eπθ [Rt:u], where Rt:u = ∑u i=0 γ
  ir(St+i, At+i), γ ∈ (0, 1) is a discount factor and Eπθ denotes the expectation operator over a policy πθ, and S1 is generated from P0. When γ = 1, to ensure the cumulative reward well defined, it is usually assumed that all policies are proper (Bertsekas & Tsitsiklis, 1996); that is, for any policy πθ, the agent goes to a recurrent state S∗ with probability 1. After the agent passes the recurrent state S∗ at a time τ , the rewards are always 0. Such a finite horizon case is called episodic MDPs (Puterman, 1994). For brevity, we denote Rt:u as R when the meaning is obvious. Under these criteria, the agent may attempt to obtain a higher cumulative reward while taking higher risks. In real-world applications, such as portfolio management in finance (Markowitz, 1952; Markowitz et al., 2000), such risky decision-making is not always desired, and we, therefore, consider the trade-off between the expected cumulative reward and the variance. Thus, while the goal of the risk-neutral MDP problem is to find the parameter θ that maximizes the total reward, we consider the mean-variance trade-off between the cumulative expected reward and the variance of the cumulative reward R in the MVRL problem. Note that even when the reward r is deterministic, the cumulative reward is a random variable owing to the stochastic policy, and there exists the mean-variance trade-off. In addition, even if the optimal policy is deterministic, the proposed method empirically has the potential to improves the stability of the training from the observations in the experiments of Section 5.3.
sections.heading: 3 EXISTING MVRL METHODS
sections.text: In this section, we introduce existing studies of MVRL.
sections.heading: 3.1 CONSTRAINED TRAJECTORY-VARIANCE PROBLEM
sections.text: Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) formulated MVRL by a constrained optimization problem defined as maxθ∈Θ Eπθ [R] s.t. Varπθ (R) ≤ η. In their formulation, the goal is to maximize the expected cumulative reward with controlling the trajectory-variance at a certain level. To solve this problem, Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) consider a penalized method defined as maxθ∈Θ Eπθ [R] − δg ( Varπθ (R) − η ) , where δ > 0 is a constant and g : R → R is a penalty function, such as g(x) = x or g(x) = x2.
sections.heading: 3.1.1 DOUBLE SAMPLING ISSUE
sections.text: Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) report the double sampling issue in MVRL, which requires sampling from two different trajectories for estimating the policy gradient. For instance, in an episodic MDP with the discount factor γ = 1 and the stop-
  ping time τ = min{t | St = S∗}, the gradients of Eπθ [R], Eπθ [ R2 ] , and ( Eπθ [R] )2 are given
  as ∇θEπθ [R] = Eπθ [R ∑τ t=1 ∇θ log πθ(St, At)], ∇θEπθ [ R2 ] = E [ R2 ∑τ t=1 ∇θ log πθ(St, At) ] ,
  and ∇θ ( Eπθ [R] )2 = 2Eπθ [R]∇θEπθ [R] (Tamar et al., 2012). Besides, the gradient of the vari-
  ance is given as Eπθ [ R2 ∑τ t=1 ∇θ log πθ(St, At) ] − 2Eπθ [R]∇θEπθ [R]. Because optimizing the policy πθ using the gradients directly is computationally intractable, we replace them with their unbiased estimators. Suppose that there is a simulator generating a trajectory k with {(Skt , Akt , r(Skt , Akt ))}τ k t=1, where τ k is the stopping time of the trajectory. Then, we can naively
  construct unbiased estimators of Eπθ [R] and Eπθ [ R2 ] as ∇̂θEπθ [R] = R̂k ∑τk t=1 ∇θ log πθ(Skt , Akt )
  and ∇̂θEπθ [ R2 ] = ( R̂k )2 ∑τk t=1 ∇θ log πθ(Skt , Akt ), where R̂k is a sample approximation
  of Eπθ [R] at the episode k. However, obtaining an unbiased estimator of ∇θ ( Eπθ [R] )2 = 2Eπθ [R]∇θEπθ [R] is difficult because it requires sampling from two different trajectories for approximating Eπθ [R] and ∇θEπθ [R]. This issue makes the optimization problem difficult.
sections.heading: 3.1.2 EXISTING SOLUTIONS TO DOUBLE SAMPLING ISSUE
sections.text: For the double sampling issue, Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) presented the following solutions.
  Multi-time-scale stochastic optimization: The proposed methods of Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) are based on stochastic approximation to find an equilibrium point of an ordinary differential equation and saddle point of the objective, respectively.
  Coordinate descent optimization: Xie et al. (2018) proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm. First, based on Lagrangian relaxation, Xie et al. (2018) set an objective function as maxθ∈Θ Eπθ [R] − δ (Varπθ (R)− η). Then, Xie et al. (2018) transformed the objective function as maxθ∈Θ,y∈R 2y ( Eπθ [R] + 12δ ) −y2−Eπθ [ R2 ] and estimated a parameter by solving the optimization problem via a coordinate descent algorithm.
  Weakness of existing approaches: The multi-time-scale approaches by Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) are known to be sensitive to the choice of the step-size schedules, which are not easy to be controlled (Xie et al., 2018). The method by Xie et al. (2018) does not reflect the constraint condition η as shown above; that is, in the objective function of Xie et al. (2018), there exits penalty coefficient δ, but does not exist the constraint condition η. Note that the problem of Xie et al. (2018) is owing to their objective function based on the penalty function g(x) = x: Eπθ [R]− δ (Varπθ (R)− η), in which the first derivative does not include η. In addition, when using quadratic function as g(x) = x2 to consider η, we cannot remove E[R2] even with Legendre-Fenchel dual; that is, the method also suffers the double sampling issue.
sections.heading: 3.2 CONSTRAINED PER-STEP VARIANCE PROBLEM
sections.text: Bisi et al. (2020) and Zhang et al. (2020) proposed solving a constrained per-step variance problem for MVRL. Bisi et al. (2020) showed that the per-step variance Varπθ (R) ≤ Varπθ (r(St,At))
  (1−γ)2 , which implies that the minimization of the per-step variance Var(r(St, At)) also minimizes trajectory-variance Varπθ (R). Therefore, they train a policy πθ by maximizing Eπθ [r(St, At)] − κVarπθ (r(St, At)), where κ > 0 is a parameter of the penalty function. The methods of Bisi et al. (2020) and Zhang et al. (2020) are based on the trust region policy optimization (Schulman et al., 2015) and coordinate descent with Legendre-Fenchel duality (Xie et al., 2018), respectively.
sections.heading: 3.3 CONSTRAINED CUMULATIVE EXPECTED REWARD PROBLEM
sections.text: While existing MVRL studies mainly focus on a constrained trajectory-variance problem, a constrained cumulative expected reward optimization is also frequently used in practical situations, such as finance (Markowitz, 1952; Markowitz et al., 2000). In the constrained cumulative expected
  reward problem, we solve the following problem:
  min θ∈Θ Varπθ (R) s.t. Eπθ [R] = ξ. (1)
  Our proposed EQUM framework is based on this motivation; that is, variance minimization. As shown in the following section, from a computational perspective, there is a critical difference between the constrained cumulative expected reward and trajectory-variance problems.
sections.heading: 4 EQUM FRAMEWORK
sections.text: In this paper, as a novel approach for MVRL, we propose a Expected Quadratic Utility maximization RL (EQUM). In economic model, by using two parameters α > 0 and β > 0, the quadratic utility function for the cumulative return R is defined as U(R;α, β) = αR − 12βR
  2 for α > 0, β ≥ 0 (Luenberger, 1998). The quadratic utility function captures the preference of a risk-averse agent over the cumulative return R. Let us consider the expected quadratic utility function defined as
  Eπθ [U(R;α, β)] = αEπθ [R]− 1
  2 βEπθ
  [ R2 ] = αEπθ [R]− 1 2 β ( Eπθ [R] )2 − 1 2 βEπθ [( Eπθ [R]−R )2] . (2)
  In the EQUM framework, we train a policy by maximizing the expected quadratic utility function.
sections.heading: 4.1 INTERPRETATIONS
sections.text: Here, we introduce four interpretations of EQUM. We can interpret the EQUM as an approach for (i) an expected utility maximization, (ii) a targeting optimization problem to achieve an expected cumulative reward ζ, (iii) a constrained trajectory-reward problem with a quadratic penalized function, and (iv) an expected cumulative reward maximization with regularization.
  First, we discuss the connection with respect to training an agent to achieve a predefined return (Berger, 1985). Let ζ be a target return that the algorithm aims to achieve. Then, we consider the mean squared error (MSE) minimization between the expected deviation of the return and ζ:
  argmin θ∈Θ J(θ; ζ) = argmin θ∈Θ
  Eπθ [( ζ −R )2] (3)
  We can decompose the MSE into the bias and variance as follows: Eπθ [( ζ −R )2] = ( ζ − Eπθ [R]
  )2︸ ︷︷ ︸ Bias +2Eπθ [( ζ − Eπθ [R] )( Eπθ [R]−R )] ︸ ︷︷ ︸
  0
  +Eπθ [( Eπθ [R]−R )2]︸ ︷︷ ︸ Variance
  = ζ2 − 2ζEπθ [R] + ( Eπθ [R] )2 + Eπθ [( Eπθ [R]−R )2] .
  Thus, the minimization problem (3) trains the policy πθ to consider the trade-off between the bias( ζ − Eπθ [R] )2 and variance Eπθ [( Eπθ [R]−R )2] (Section 9.5 of Luenberger (1998)). Moreover, we find that the EQUM is equivalent to the reward-targeting optimization when ζ = αβ ; that is,
  argmin θ∈Θ J
  ( θ; α
  β
  ) = argmin
  θ∈Θ
  ( α
  β
  )2 − 2α
  β Eπθ [R] +
  ( Eπθ [R] )2 + Eπθ [( Eπθ [R]−R )2] = argmax
  θ∈Θ Eπθ [U(R;α, β)] .
  Second, the bias-variance trade-off heuristically provides a solution to a constraint optimization problem (1) with the constraint ξ = ζ = αβ by solving the following penalized problem:
  min θ∈Θ
  Varπθ (R) + ( Eπθ [R]− ξ )2 . (4)
  Third, we can regard the quadratic utility function as an expected cumulative reward maximization with a regularization term defined as E [ R2 ] ; that is, minimization of the risk R(πθ):
  R(θ) = −Eπθ [R]︸ ︷︷ ︸ Risk of expected cumulative reward maximization
  + ψEπθ [ R2 ]︸ ︷︷ ︸ Regularization term
  (5)
  where ψ > 0 is a regulation parameter and ψ = β2α = 1 2ζ . As ζ → ∞, R(πθ) → −Eπθ [R].
sections.heading: 4.2 MERITS OF THE EQUM FRAMEWORK
sections.text: In this section, we present two advantages of the EQUM framework. The first advantage is in computation. The EQUM framework is an MVRL method. However, compared with existing MVRL methods, which involve the double sampling issue, the computation of the EQUM framework is much simpler because we transform the MVRL problem (4) into the optimization problem without the term (Eπθ [R])
  2 (see (2) and (5)). The second advantage is that it provides a variety of interpretations. Because the EQUM framework can interpret economic theory, it is applicable for modeling economic dynamics. In addition, as one of the MVRL methods, it is suitable for various real-world applications, such as finance (Deng et al., 2016) and playing games. By regarding the proposed EQUM as a regularization framework, we can combine it with existing RL methods.
sections.heading: 4.3 IMPLEMENTATIONS OF THE EQUM FRAMEWORK
sections.text: Here, we discuss the implementations of the EQUM framework.
  Simplest policy gradient with EQUM With the EQUM framework, we introduce a main algorithm with simplest policy gradient (SPG) algorithm (Brockman et al., 2016), which is also called the REINFORCE algorithm (Williams, 1992). For an episode k with length n, the method replaces the expectations Eπθ [ R ] and Eπθ [ R2 ] with the sample approximations ∑n t=1 γ
  t−1r(St, At) and(∑n t=1 γ t−1r(St, At) )2 , respectively (Brockman et al., 2016). Then, the unbiased gradients are
  ∇̂θEπθ [R] = R̂k n∑ t=1 ∇θ log πθ(Skt , Akt ) and ∇̂θEπθ [ R2 ] = ( R̂k )2 n∑ t=1 ∇θ log πθ(Skt , Akt ).
  Therefore, for a sample approximation R̂k of Eπθ [ R2 ] at the episode k, we optimize the policy with
  ascending the following gradient: ( αR̂k − 12β ( R̂k )2)∑n t=1 ∇θ log πθ(Skt , Akt ).
  Actor-critic with EQUM: For another combination with the EQUM framework, we apply an actor-critic (AC) based algorithms, which is also refereed to as the advantage actor-critic (A2C) algorithm (Williams & Peng, 1991; Mnih et al., 2016). Extending the AC algorithm, for an episode k with the length n, we train the policy by a gradient defined as
  ∇θ log πθ(Skt , Akt ) {( αR̃kt:t+n−1 − 1
  2 βR̃k,2t:t+n−1
  ) − ( αM (1)
  ω̂ (1) k
  (Skt )− 1
  2 βM
  (2) ω̂ (2) k (Skt )
  )} ,
  where R̃kt:t+n−1 = R̂ k t:t+n−1+γ nM (1)
  ω̂ (1) k
  (Skt+n), and R̃ k,2 t:t+n−1 = ( R̂kt:t+n−1 + γ nM (1)
  ω̂ (1) k
  (Skt+n)
  )2 ,
  and M (1) ω̂
  (1) k
  (Skt ) and M (2)
  ω̂ (2) k
  (Skt ) are value functions approximating E[Rt+1:∞] and E[R2t+1:∞] with
  parameters ω̂(1)k and ω̂ (2) k , respectively.
  For other RL algorithms, we can heuristically extend our proposed framework EQUM to accept the other RL algorithms by adding E[R2] as regularization term.
  Determining α and β: Next, we discuss the parameter tuning of α and β, which are equivalent to ζ, ξ, and ψ). As explained, the parameter ψ has several meanings based on the interpretations of the EQUM, such as the quadratic utility function, targeting optimization, and constrained optimization. In addition, from the regularization perspective, we can adjust the parameter
  to maximize the expected cumulative reward in the validation data. Thus, we propose the following four directions for the parameter tuning. First, in economic applications, such as finance, we choose β2α = ψ based on the theoretical economic assumptions and economic empirical studies (Ziemba et al., 1974; Kallberg & Ziemba, 1983) (Appendix A). For instance, Capital Asset Pricing Model (CAPM), which is one of the most popular economic models, is also base on the quadratic utility function (Sharpe, 1964; Lintner, 1965; Mossin, 1966). Second, we set ζ = 12ψ as the targeted reward to achieve. Third, we regard the parameter ψ as the constrained problem (1). Fourth, through cross-validation, we optimize the regularization parameter ψ.
sections.heading: 5 EXPERIMENTS
sections.text: This section investigates the empirical performance of the proposed EQUM with synthetic and realworld financial datasets. The goal of these experiments is to construct a portfolio with a wellcontrolled mean and variance through algorithms. In addition, we also show experimental results using CartPole and Atari games. Note that the per-step rewards of portfolio selection are stochastic variables and those of CartPole and Atari games are deterministic variables. However, even when the reward is deterministic, the cumulative reward has randomness owing to the stochastic policy.
sections.heading: 5.1 EXPERIMENTS WITH SYNTHETIC PORTFOLIO SELECTION DATASETS
sections.text: Following Tamar et al. (2012) and Xie et al. (2018), we artificially generate a portfolio dataset. Let us consider a portfolio composed of two types of assets: a liquid asset, which has a fixed interest rate rl = 1.001, and a non-liquid asset, which has a time-dependent interest rate rnl(t) ∈ {1.1, 2}. While we can sell the liquid asset at every time step t = 1, 2, . . . , T , we can sell the non-liquid asset only after a maturity period of N steps; that is, when holding 1 liquid asset, we obtain 1.001 per period; when holding 1 non-liquid asset at the t-th period, we obtain 1.1 or 2 at the t+N -th periods. In addition, the non-liquid asset has some risk of not being paid with a probability prisk; that is, if the non-liquid asset defaulted during the N periods, we cannot obtain any profits by having the asset. In this setup, a typical investment strategy is to construct a portfolio using both liquid and non-liquid assets to control the mean and variance. In our model, the investor may change his portfolio by investing a fixed fraction α = 0.2 to the non-liquid asset at each time step. As a performance metric of the portfolio, we focus on the mean and variance of the cumulative reward when having the cash 1 at the first period and investing the cash for 50 periods following an algorithm.
  In particular, we aim to investigate the sensitivity of the EQUM against the parameter ψ = 2/ζ and compare the EQUM with the standard SPG and AC algorithms. We apply the SPG algorithm with the EQUM (Section 4.3) to the synthetic datasets. For ζ, we use ζ = 1, 2, 3, 4, 5, 6. Note that from the regularization perspective with a parameter ψ, EQUM with ψ = 0.1 is equal to minimize the MSE between the cumulative reward and ζ = 5 = 1/(2 × 0.1). We train the model with 500 episodes. In Figure 1, we calculate the average reward and standard deviation at each episode of the training process over 1, 000 trials. In Table 1, using the trained model for the test environment, we show the average reward (AR) and standard deviation (SD) by conducting 1, 000 trials.
  For each trial, we compute the AR and SD over 100 trial and compute the average of each AR and SD over 1, 000 trials. From Figure 1 and Table 1, we can clearly confirm that the EQUM framework
  realize the control of the mean-variance trade-off. As shown in the results, we can reduce the variance by increasing ζ. The MSE from target ζ is shown in Table 2. In Figure 2, for the EQUM with various ζ and the method of Tamar et al. (2012) (Tamar) with various variance constraints (var), we plot the AR and variance (Var) computed on the train environment over 1, 000 trials. We show the Var instead of the SD because the Tamar controls the variance. Compared with Tamar, the EQUM returns Pareto efficient portfolios; that is, higher AR and lower Var. We consider unlike the direct optimization of the EQUM, the Tamar suffers from the complicated optimization mechanism. Figure 3 in Appendix B.1 corresponds to the test environment version of Figure 2.
  In Appendix B.2, we also show experimental results using another synthetic dataset of Americanstyle option following Tamar et al. (2014) and Xie et al. (2018).
sections.heading: 5.2 EXPERIMENTS WITH REAL-WORLD DATASET
sections.text: We use well-known benchmarks called Fama & French (1992) (FF) datasets to ensure the reproducibility of the experiment. We use the FF25, FF48 and FF100 datasets. The FF25 and FF 100 dataset includes 25 and 100 portfolios formed based on size and book-to-market ratio; the FF48 dataset contains 48 portfolios representing different industrial sectors. We formulate the problem by an episodic MDP. We use all datasets covering monthly data from July 1980 to June 2020. The state is past 12 months returns of each asset and the action is defined as portfolio weight; that is, the number of actions is equal to that of assets. The reward is obtained as the portfolio return. Here, the portfolio return at time 1 ≤ t ≤ T is defined as yt = ∑m j=1 yj,twj,t−1, where yj,t is the return of j asset at time t, wj,t−1 is the weight of j asset in the portfolio at time t − 1, and m is the number of assets. The length of the episode is 1 years (12 months). For the stochastic policy, we adopt a three-layer feed-forward neural network with the ReLU activation function where the number of units in each respective layer is equal to the number of assets, 100, 50, and the number of actions. We use the softmax function for the output layer.
  Portfolio models: We use the following portfolio models. An equally-weighted portfolio (EW) weights the financial assets equally (DeMiguel et al., 2007). A mean-variance portfolio (MV) computes the optimal variance under a mean constraint (Markowitz, 1952). For computing the mean vector and covariance matrix, we use the latest 10 years (120 months) data. An Kelly growth optimal portfolio with ensemble learning (EGO) is proposed by Shen et al. (2019). We set the number of resamples as m1 = 50, the size of each resample m2 = 5τ , the number of periods of return data τ = 60, the number of resampled subsetsm3 = 50, and the size of each subsetm4 = n0.7, wherem is number of assets; that is, m = 25 in FF25, m = 48 in FF48 and m = 100 in FF100. A portfolio blending via Thompson sampling (BLD) is proposed by Shen & Wang (2016). We use the latest 10 years (120 months) data to compute for the sample covariance matrix and blending parameters. A policy gradients with variance related risk criteria (Tamar) is proposed by Tamar et al. (2012). We set the target variance terms η as 250,500,1000. A block coordinate ascent algorithm proposed by Xie et al. (2018), which is referred to as Mean-Variance Portfolio (MVP). We set the regularization parameters δ as 10,100,1000. Then, let us denote the SPG with the EQUM framework as EQUM. The parameter ψ is chosen from 1/3, 2/3, and 1. For optimizing Tamar, MVP and EQUM, we set the Adam optimizer with learning rate 0.01 and weight decay parameter 0.1. We train the neural networks for 10 episodes. Each portfolio is updated by sliding one-month-ahead.
  Performance metrics: The following measures widely used in finance to evaluate portfolio strategies (Brandt, 2010) are chosen. The cumulative return (CR), annualized risk as the standard de-
  MaxDD↓ 0.55 0.72 0.58 0.36 0.39 0.49 0.38 0.52 0.65 0.46 0.24 0.31 0.39 viation of return (RISK) and risk-adjusted return (R/R) are defined as follows: CR = ∑T t=1 yt,
  RISK = √
  12 T−1 ∑T t=1(yt − CR/T )2, and R/R = 12 T ×CR/RISK. R/R is the most important mea-
  sure for a portfolio strategy. We also evaluate the maximum draw-down (MaxDD), which is another widely used risk measure Magdon-Ismail & Atiya (2004) for the portfolio strategy. In particular, MaxDD is the largest drop from a peak defined as MaxDD = mint∈[1,T ] ( 0, Wtmaxτ∈[1,t]Wτ − 1 ) ,
  where Wk is the cumulative return of the portfolio until time k; that is, Wt = ∏t t′=1(1 + yt′).
  Table 3 reports the performances of the portfolios. In almost all cases, the EQUM portfolio achieves the highest R/R and the lowest MaxDD. Therefore, we can confirm that the EQUM portfolio has a high R/R, and avoids a large drawdown. The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP, and EQUM is shown in Appendix B.3. Except for FF48’s MVP, the objective itself is smaller than EQUM’s. Since the values of the objective is the same as the RR, we can empirically confirm that the better optimization, the better performance.
sections.heading: 5.3 EXPERIMENTS WITH CARTPOLE AND ATARI GAMES
sections.text: We also conduct experiments using CartPole and Atari games, where the reward is given as deterministic, and the randomness of the cumulative reward depends only on the stochastic policy. We investigate the sensitivity of ψ = 1/(2ζ) with CartPole and the compared the performance of the EQUM framework with that of Tamar et al. (2012) and Xie et al. (2018). The results are shown in Appendix B.4. In many experimental results, contrary to our expectations, we observed that the EQUM also improves the expected cumulative reward, not only the variance. We hypothesize that this is because there is often a limit on the cumulative rewards achieved by the standard expected cumulative reward maximizing algorithms. For instance, when the reward at each period is 1, and the discount factor is 0.99, the infinite sum is 100. In such a case, instead of naive reward maximization, MSE minimization against the target reward 100 may result in a more stable performance empirically. We also hypothesize that even if an optimal policy is deterministic, the EQUM can improve the stability of the training process by reducing the variance. This observation is an open problem related to exploration and exploitation trade-off. Therefore, unless we could increase the cumulative reward infinitely, the proposed EQUM framework can stabilize the performance. From this aspect, we can confirm that the EQUM provides a regularization effect.
sections.heading: 6 CONCLUSION
sections.text: In this paper, we proposed an EQUM framework as a variant of MVRL. Compared with existing MVRL methods, the EQUM framework is computationally friendly. The proposed EQUM framework also includes various interpretations, such as targeting optimization and regularization and is suitable for many real-world applications, such as finance and playing games. We investigated the effectiveness of the EQUM framework compared with the standard RL and existing MVRL methods through the experiments. In the results, the proposed method successfully controls the mean-variance trade-off. As an open problem, we also observed that even when an optimal policy is deterministic, the proposed method improves performance. We hypothesize that the proposed method affects the stabilization of the training process.
sections.heading: A ECONOMIC THEORY AND QUADRATIC UTILITY FUNCTION
sections.text: Considering the mean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al. (2012) and Xie et al. (2018) pointed out. The mean-variance trade-off is justified by assuming either quadratic utility function to the economic agent or multivariate normal distribution to the financial assets. By assuming the quadratic utility function or the normal distribution, we can assume that the expected utility function of the agent is maximized by maximizing the expected reward and minimizing the variance. Based on this observation, Markowitz (1952) proposed the following steps for providing a portfolio to an economic agent (Also see Markowitz (1959), page 288, and Luenberger (1998)):
  • Constructing portfolios minimizing the variance under several reward constraint; • Among the portfolios constructed in the first step, the economic agent chooses a portfolio
  maximizing the utility function.
  Therefore, the goal of Markowitz’s portfolio is not only to construct the portfolio itself but also to maximize the expected utility function of the agent. In traditional economics, this two-step is adopted because directly predicting the reward and variance to maximize the expected utility function is difficult; therefore, first gathering information based on analyses of an economist, then we construct the portfolio using the information and provide the set of the portfolios to an economic agent. However, owing to the recent development of machine learning, we can directly represent the complicated economic dynamics using flexible models, such as deep neural networks. In addition, as Tamar et al. (2012) and Xie et al. (2018) reported, when constructing the mean-variance portfolio in RL, we suffer the double sampling issue. Therefore, in this paper, we aim to achieve the original goal of the mean-variance approach; that is, the expected utility maximization. Note that this idea is not restricted to financial applications but can be applied to applications where the agent utility can be represented only by the mean and variance. In the following subsection, we review the existing studies of finance and quadratic utility function.
  A.1 MARKOWITZ’S PORTFOLIO AND CAPITAL ASSET PRICING MODEL
  Markowitz’s portfolio is known as the mean-variance portfolio (Markowitz, 1952; Markowitz et al., 2000). Constructing the mean-variance portfolio is motivated by the agent’s expected utility maximization. When the utility function is given as the quadratic utility function, or the financial asset follows the multivariate normal distribution, a portfolio maximizing the agent’s expected utility function is given as a portfolio with minimum variance under a certain standard expected reward.
  The Capital Asset Pricing Model (CAPM) theory is a concept which is closely related to Markowitz’s portfolio (Sharpe, 1964; Mossin, 1966; Lintner, 1965). This theory theoretically explains the expected return of investors when the investor invests in a financial asset; that is, it derives the optimal price of the financial asset. To derive this theory, as well as Markowitz’s portfolio, we assume the quadratic utility function to the investors or the multivariate normal distribution to the financial assets.
  Merton (1969) extended the static portfolio selection problem to a dynamic case. Fishburn & Porter (1976) studied the sensitivity of the portfolio proportion when the safe and risky asset distributions change under the quadratic utility function. Thus, there are various studies investigating relationship between the utility function and risk-averse optimization (Tobin, 1958; Kroll et al., 1984; Bulmuş & Özekici, 2014; Bodnar et al., 2015).
  A.2 EMPIRICAL STUDIES ON THE UTILITY FUNCTIONS
  The standard financial theory is built on the assumption that the economic agent has the quadratic utility function. For supporting this theory, there are several empirical studies to estimate the parameters of the quadratic utility function. Ziemba et al. (1974) investigated the change of the portfolio proportion when the parameter of the quadratic utility function changes using the Canadian financial dataset. Recently, Bodnar et al. (2018) investigate the risk parameter (α and β in our formulation of the quadratic utility function) using the markets indexes in the world. They found that the utility function parameter depends on the market data model.
  A.3 CRITICISM
  For the simple form of the quadratic utility function, the financial models based on the utility are widely accepted in practice. However, there is also criticism that the simple form cannot capture the real-world complicated utility function. For instance, Kallberg & Ziemba (1983) criticized the use of the quadratic utility function and proposed using a utility function, including higher moments. This study also provided empirical studies using U.S. financial dataset for investigating the properties of the alternative utility functions. However, to the best of our knowledge, financial practitioners still use financial models based on the quadratic utility function. We consider this is because the simple form gains the interpretability of the financial models.
sections.heading: B DETAILS OF EXPERIMENTS
sections.text: In this section, we show the additional experiments and describe the details of the experiments. To implement SPG and AC algorithms, we mainly follow the Pytorch example1.
  B.1 PARETO EFFICIENCY ON THE TEST ENVIRONMENT OF THE SYNTHETIC PORTFOLIO SELECTION DATASET
  For investigating the Pareto efficiency, we show the AR and Var of the train environment in Figure 2 of Section 5.1. Here, we also show the AR and Var of the test environment in Figure 3.
  B.2 EXPERIMENTS USING THE SYNTHETIC AMERICAN-STYLE OPTION DATASET
  Among various options in finance, an American-style option refers to a contract that we can execute an option right at any time before the maturity time τ ; that is, a buyer who bought a call option has a right to buy the asset with the call option strike price Wcall at any time; a buyer who bought a put option has a right to sell the with the call option strike price Wput at any time.
  In the setting of Tamar et al. (2014) and Xie et al. (2018), the buyer simultaneously buy call and put options, which have the strike price Wcall = and Wput =, respectively. The maturity time is set as τ =. If the buyer executes the option at time t, the buyer obtains a reward rt = max(0,Wput − xt)+max(0, xt−Wcall), where xt is an asset price. We set x0 = 1 and define the stochastic process as follows: xt = xt−1fu with probability 1 and xt = xt−1fd with probability 1, where fu and fd. These parameters follows Xie et al. (2018). Under this setting, we investigate the performance of our EQUM. For the EQUM, we use ζ = 0.1, 0.3, 0.5, 0.7, 1, 1.3. For the other settings, we follow the previous experiment for portfolio management.
  1https://github.com/pytorch/examples/tree/master/reinforcement_learning
  In Figure 4, we calculate the average reward and standard deviation at each episode of the training process by conducting 1, 000 trials. In Table 4, using the trained model for the test environment, we show the average reward (AR) and standard deviation (SD) by conducting 1, 000 trials. We show the MSE between the realized reward and the target reward ζ in Table 5, where the lowest MSE method is highlighted in bold. As well as the experimental results with the synthetic portfolio dataset, we can confirm that the EQUM can control the risk well. In addition, as well as the experiments with CartPole and Atari games, we can observe that the EQUM also increases the expected reward contrary to our expectation.
  B.3 DETAILS OF EXPERIMENTS OF PORTFOLIO OPTIMIZATION
  The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP and EQUM is shown in Table 6. Except for FF48’s MVP, the objective itself is smaller than EQUM’s. Since the values of objective is the same as the RR, we can empirically confirm that the better optimization, the better performance.
  We also divide the performance period into two for robustness checks. Table 7 shows the first-half results from July 2000 to June 2010 and the second-half results from July 2010 to June 2020. In almost all cases, the EQUM portfolio achieves the highest R/R.
  B.4 DETAILS OF EXPERIMENTS WITH CARPOLE AND ATARI GAMES
  In this section, we report the experimental performances of the proposed EQUM framework with well-known benchmarks. We investigate how the behaviors of existing RL methods are changed by adding the additional E[R2] term. We use a simple two-layer perceptron for modeling the policy following the Pytorch example (Paszke et al., 2019). For the SPG-based algorithms, we define the cumulative reward as finite sum with γ = 1 following (Tamar et al., 2012). For the AC-based algorithms, we define the cumulative reward as infinite sum with γ = 0.99 following (Prashanth & Ghavamzadeh, 2013). For the SPG-based algorithms, we define the cumulative reward as finite sum with γ = 1 following (Tamar et al., 2012). For the AC-based algorithms, we define the cumulative reward as infinite sum with γ = 0.99 following (Prashanth & Ghavamzadeh, 2013).
  B.4.1 SENSITIVITY ANALYSIS ON ψ
  First, we investigate the sensitivity of the EQUM framework to the parameter ψ using the CartPole. We apply the SPG and AC methods (Section 4.3) with the EQUM framework to the Cartpole problem, which is a classic control problem. We use ψ = 0.001, 0.002, 0.003, 0.005, 0.01, 0.1 and compare the results with the standard SPG and AC methods. For instance, from the targeting optimization perspective, EQUM with ψ = 0.001 is equal to minimize the MSE between the cumulative reward and ζ = 500 = 1/(2 ∗ 0.001). We train the model by 300 episodes. We calculate the average reward and standard deviation at each period by conducting 300 trials. The results are shown
  in Figure 5. In the SPG results, we can confirm the mean-variance trade-off. In contrast, in the SPG with ψ = 0.001 and all AC results, the EQUM framework improves the expected cumulative reward. These results are contrary to our expectations because MVRL methods typically decrease the expected cumulative return to decrease the variance. We discuss this topic in more detail in Section 5.3.
  B.4.2 EXPERIMENTS OF ATARI GAMES
  When playing games, we often consider risk-control while maintaining a certain level of reward. Under this motivation, we benchmark the proposed EQUM framework on four Atari game tasks from the OpenAI gym (Brockman et al., 2016). Among the games, we choose BeamRider, Seaquest, Qbert, and SpaceInvaders in which SPG and AC methods work well. We use simplified environments in which the observations are the RAM of the Atari machine, consisting of only 128 bytes. We compare our methods with the standard SPG, and MVLR methods of Tamar et al. (2012) and Xie et al. (2018). We calculate the average reward (AR) at the last episode over 5 trials and the standard deviation (SD). In Table 8, we show the results of the SPG algorithm with the standard setting and EQUM framework, where ψ is chosen from 0.001, 0.003, 0.005, 0.01, 0.03, and 0.3. Note that when using ψ = 0.005, it is equal to minimize the MSE between the expected cumulative reward and the target 100. In almost all cases, the EQUM framework shows the better AR than the standard methods. As well as the previous sensitivity experiment, this result is contrary to our expectation. Among the games, we choose simplified BeamRider, Seaquest, Qbert, and SpaceInvaders in which the SPG algorithm work well, and the observations are the RAM of the Atari machine, consisting of only 128 bytes. We compare our methods with the standard SPG algorithms, and MVRL methods of Tamar et al. (2012) and Xie et al. (2018). We denote the SPG-based methods proposed by Tamar et al. (2012) and Xie et al. (2018) as Tamar and MVP, respectively. We calculate the average reward (AR) at the last episode over 5 trials and the standard deviation (SD). We choose the parameter of Tamar from 200 and 2000, which is denoted as b in Tamar et al. (2012). We choose the parameter of MVP from 1 and 10, which is denoted as λ in Tamar et al. (2012). The parameters are denoted as param in Table 8.