summaries: The paper proposes an Expected Quadratic Utility Maximization (EQUM) framework for policy grandient Mean-Variance control. The authors claim that current state-of-the-art methods suffer either suffer from computational issues or cannot control risk at desiderable level, hence they propose their approach as a possible solution. They provide different interpretation for the EQUM: standard objective with a regularization, variance minimization with a constraint on expected return and a return targeting optimization. A policy gradient algorithm for EQUM is proposed together with its Actor-Critic extension.

summaries: The paper proposes a policy gradient style RL algorithm that optimizes an expected quadratic utility, a commonly used objective of risk management in finance and economics. The key idea here is based on the observation that when using the quadratic utility function, the use of mean-variance RL methods can be shown to optimize the utility of the agent. To this effect, the paper considers the use of expected quadratic utility maximization in the policy gradient. The quadratic utility can be naturally modeled using mean and variance. The paper implements two variations -- policy gradient and actor-critic with EQUM framework.

summaries: In this paper the author proposed a new mean-variance algorithm whose policy gradient algorithm is more simpler than other SOTA methods and it has an unbiased gradient. Instead of formulating the problem as an traditional mean variance constrained problem, the authors utilized quadratic utility theory and formulate the problem as variance minimization problem with a mean reward equality constraint. Then by reformulating the problem with the penalized problem and opening up the variance formulation, they showed that this mean-variance formulation does indeed have an unbiased policy gradient, that does not require advanced techniques such as double sampling or frenchel duality. To demonstrate the effectiveness of this method on balancing risk and return, they also evaluate their methods on several risk-sensitive RL benchmarks (such as portfolio optimization) and compared with a wide range of risk-sensitive RL methods.

abstractText: In real-world decision-making problems, risk management is critical. Among various risk management approaches, the mean-variance criterion is one of the most widely used in practice. In this paper, we suggest expected quadratic utility maximization (EQUM) as a new framework for policy gradient style reinforcement learning (RL) algorithms with mean-variance control. The quadratic utility function is a common objective of risk management in finance and economics. The proposed EQUM has several interpretations, such as reward-constrained variance minimization and regularization, as well as agent utility maximization. In addition, the computation of the EQUM is easier than that of existing mean-variance RL methods, which require double sampling. In experiments, we demonstrate the effectiveness of the EQUM in benchmark setting of RL and financial data.

authors:
Jonathan Baxter
Peter L. Bartlett

title: Infinite-horizon policy-gradient estimation

venue: Journal of Artificial Intelligence Research,

title: Statistical Decision Theory and Bayesian Analysis

authors:
Lorenzo Bisi
Luca Sabbioni
Edoardo Vittori
Matteo Papini
Marcello Restelli

title: Risk-averse trust region optimization for reward-volatility reduction

venue: In IJCAI,

authors:
Taras Bodnar
Nestor Parolya
Wolfgang Schmid

title: On the Exact Solution of the Multi-Period Portfolio Choice Problem for an Exponential Utility under Return Predictability

venue: European Journal of Operational Research,

authors:
Taras Bodnar
Yarema Okhrin
Valdemar Vitlinskyy
Taras Zabolotskyy

title: Determination and estimation of risk aversion coefficients

venue: Computational Management Science,

authors:
Stephen Boyd
Lieven Vandenberghe

title: Convex Optimization

title: Portfolio choice problems

venue: In Handbook of financial econometrics: Tools and techniques

authors:
T. Bulmuş
S. Özekici

title: Portfolio selection with hyperexponential utility functions

venue: OR Spectrum,

authors:
Yinlam Chow
Mohammad Ghavamzadeh

title: Algorithms for cvar optimization in mdps

venue: In NeurIPS

authors:
Yinlam Chow
Mohammad Ghavamzadeh
Lucas Janson
Marco Pavone

title: Risk-constrained reinforcement learning with percentile risk criteria

venue: Journal of Machine Learning Research,

authors:
Victor DeMiguel
Lorenzo Garlappi
Raman Uppal

title: Optimal versus naive diversification: How inefficient is the 1/n portfolio strategy

venue: The review of Financial studies,

authors:
Yue Deng
Feng Bao
Youyong Kong
Zhiquan Ren
Qionghai Dai

title: Deep direct reinforcement learning for financial signal representation and trading

venue: IEEE Transactions on Neural Networks and Learning Systems,

authors:
Eugene F Fama
Kenneth R French

title: The cross-section of expected stock returns

venue: Journal of Finance,

authors:
Peter C. Fishburn
R. Burr Porter

title: Optimal portfolios with one safe and one risky asset: Effects of changes in rate of return and risk

venue: Management Science,

authors:
Javier García
Fernando Fernández

title: A comprehensive survey on safe reinforcement learning

authors:
Peter Geibel
Fritz Wysotzki

title: Risk-sensitive reinforcement learning applied to control under constraints

authors:
J.G. Kallberg
W.T. Ziemba

title: Comparison of alternative utility functions in portfolio selection problems

authors:
Yoram Kroll
Haim Levy
Harry Markowitz

title: Mean-variance versus direct utility maximization

title: The valuation of risk assets and the selection of risky investments in stock portfolios and capital budgets

venue: The Review of Economics and Statistics,

title: Portfolio Selection: Efficient Diversification of Investments

title: Portfolio selection

venue: The journal of finance,

authors:
H.M. Markowitz
G.P. Todd
W.F. Sharpe

title: Mean-Variance Analysis in Portfolio Choice and Capital Markets

title: Lifetime portfolio selection under uncertainty: The continuous-time case

authors:
Volodymyr Mnih
Adria Puigdomenech Badia
Mehdi Mirza
Alex Graves
Timothy Lillicrap
Tim Harley
David Silver
Koray Kavukcuoglu

title: Asynchronous methods for deep reinforcement learning

title: Equilibrium in a capital asset

venue: market. Econometrica,

authors:
Adam Paszke
Sam Gross
Francisco Massa
Adam Lerer
James Bradbury
Gregory Chanan
Trevor Killeen
Zeming Lin
Natalia Gimelshein
Luca Antiga
Alban Desmaison
Andreas Kopf
Edward Yang
Zachary DeVito
Martin Raison
Alykhan Tejani
Sasank Chilamkurthy
Benoit Steiner
Lu Fang
Junjie Bai
Soumith Chintala

title: Pytorch: An imperative style, high-performance deep learning library

venue: In NeurIPS,

authors:
L.A. Prashanth
Mohammad Ghavamzadeh

title: Actor-critic algorithms for risk-sensitive mdps

title: Variance-constrained actor-critic algorithms for discounted and average reward mdps

venue: Machine Learning,

title: Markov Decision Processes: Discrete Stochastic Dynamic Programming

authors:
John Schulman
Sergey Levine
Pieter Abbeel
Michael Jordan
Philipp Moritz

title: Trust region policy optimization

venue: In ICML,

title: Capital asset prices: A theory of market equilibrium under conditions of risk

venue: The Journal of Finance,

authors:
Weiwei Shen
Jun Wang

title: Portfolio blending via thompson sampling

authors:
Weiwei Shen
Bin Wang
Jian Pu
Jun Wang

title: The kelly growth optimal portfolio with ensemble learning

venue: In AAAI,

authors:
Richard S. Sutton
Andrew G. Barto

title: Reinforcement Learning: An Introduction

authors:
Richard S. Sutton
David McAllester
Satinder Singh
Yishay Mansour

title: Policy gradient methods for reinforcement learning with function approximation

venue: NeurIPS,

authors:
Aviv Tamar
Dotan Di Castro
Shie Mannor

title: Policy gradients with variance related risk criteria

authors:
Aviv Tamar
Shie Mannor
Huan Xu

title: Scaling up robust mdps using function approximation

title: Liquidity Preference as Behavior Towards Risk

venue: Review of Economic Studies,

title: Toward a Theory of Reinforcement-learning

venue: Connectionist Systems

authors:
Ronald Williams
Jing Peng

title: Function optimization using connectionist reinforcement learning algorithms

venue: Connection Science,

title: Simple statistical gradient-following algorithms for connectionist reinforcement learning

authors:
Tengyang Xie
Bo Liu
Yangyang Xu
Mohammad Ghavamzadeh
Yinlam Chow
Daoming Lyu
Daesub Yoon

title: A block coordinate ascent algorithm for mean-variance optimization

venue: NeurIPS

authors:
Shangtong Zhang
Bo Liu
Shimon Whiteson

title: Mean-variance policy iteration for risk-averse reinforcement learning, 2020

authors:
W.T. Ziemba
C. Parkan
R. Brooks-Hill

title: Calculation of investment portfolios with risk free borrowing and lending

title: A ECONOMIC THEORY AND QUADRATIC UTILITY FUNCTION Considering the mean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al

authors: Wcall Wput respectively

title: SPG-based methods proposed by Tamar et al

venue: We choose the parameter of MVP from 1 and 10,

heading: 1 INTRODUCTION

text:
Reinforcement learning (RL) with Markov decision processes (MDPs) is one type of dynamic decision-making problem (Puterman, 1994; Sutton & Barto, 1998). While the typical objective is the expected cumulative reward maximization, risk-aware decision-making has attracted great attention in real-world applications, such as finance and robotics (Geibel & Wysotzki, 2005; García & Fernández, 2015). The notion of risk is related to the fact that even an optimal policy may perform poorly owing to the stochastic nature of the problem. To capture the risk, various criteria have been proposed, such as Value at Risk (Luenberger, 1998; Chow & Ghavamzadeh, 2014; Chow et al., 2017) and variance (Markowitz, 1952; Markowitz et al., 2000; Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013). Among them, we focus on the mean-variance trade-off.
Typical mean-variance RL (MVRL) methods attempt to maximize the expected cumulative reward while maintaining the variance threshold (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013; 2016; Xie et al., 2018; Bisi et al., 2020; Zhang et al., 2020). However, most existing MVRL methods suffer from high computational costs owing to the double sampling issue when approximating the gradient of the variance term (Tamar et al., 2012; Prashanth & Ghavamzadeh, 2013; 2016). To avoid the double sampling issue, Xie et al. (2018) proposed a method based on the Legendre-Fenchel duality (Boyd & Vandenberghe, 2004). Although the method does not suffer from the double sampling issue, we cannot apply a standard policy gradient method and must use a coordinate descent algorithm. In addition, the method cannot control risk at a certain desirable level.
From an economics perspective, the difference in the RL objectives arises from the forms of utility functions. When the objective of an agent is expected cumulative reward maximization, the utility function is risk-neutral; when an agent attempts to control the risk based on an expected reward, the utility function is risk-averse (Mas-Colell et al., 1995). In economics, there have been several risk-averse utility functions proposed. The quadratic utility function is one such functions and is frequently used in financial economics (Luenberger, 1998). Under the quadratic utility function, the mean-variance portfolio maximizes the utility of the investor. In addition, other various financial theories are also based on the quadratic utility maximization (Markowitz, 1952; Sharpe, 1964; Lintner, 1965; Mossin, 1966). For more details, see Appendix A. In this study, as one of the MVRL approaches, we consider the expected quadratic utility maximization (EQUM) based on the policy gradient method (Williams, 1988; 1992; Sutton et al., 1999; Baxter & Bartlett, 2001).
The EQUM has the following advantages: (i) low computational cost; (ii) numerous interpretations, and (iii) direct connections to real-world applications. In this study, as interpretations of EQUM, we propose the minimization of the variance under the constraint of the expected cumulative reward, reward-targeting optimization, and regularization. Thus, this study contributes to the context of riskaverse RL and MVRL. In the following sections, we first formulate the problem setting in Section 2 and review the existing methods in Section 3. Then, we propose the main algorithms in Section 4. Finally, we investigate the empirical effectiveness of the EQUM in Section 5.

heading: 2 PROBLEM SETTING

text:
We consider the standard RL framework, where a learning agent interacts with an unfamiliar, dynamic, and stochastic environment modeled by a Markov decision process (MDP) in discrete time. We define the MDP as the tuple (S,A, r, P, P0), where S is a set of states, A is a set of actions, r : S × A → R is a reward function, P : S × S × A → [0, 1] is the transition kernel, and P0 : S → [0, 1] is an initial state distribution. The initial state S1 is sampled from P0. Let πθ : A × S → [0, 1] be a parameterized stochastic policy mapping states to actions, where θ is the tunable parameter. At time step t, an agent chooses an actionAt according to a policy πθ(· | St). We assume that the policy πθ is a differentiable function with respect to θ; that is, ∂πθ(a,s) ∂θ exists.
There are several performance measures for a policy πθ. One popular measure is the expected cumulative reward from time step t to u defined as Eπθ [Rt:u], where Rt:u = ∑u i=0 γ
ir(St+i, At+i), γ ∈ (0, 1) is a discount factor and Eπθ denotes the expectation operator over a policy πθ, and S1 is generated from P0. When γ = 1, to ensure the cumulative reward well defined, it is usually assumed that all policies are proper (Bertsekas & Tsitsiklis, 1996); that is, for any policy πθ, the agent goes to a recurrent state S∗ with probability 1. After the agent passes the recurrent state S∗ at a time τ , the rewards are always 0. Such a finite horizon case is called episodic MDPs (Puterman, 1994). For brevity, we denote Rt:u as R when the meaning is obvious. Under these criteria, the agent may attempt to obtain a higher cumulative reward while taking higher risks. In real-world applications, such as portfolio management in finance (Markowitz, 1952; Markowitz et al., 2000), such risky decision-making is not always desired, and we, therefore, consider the trade-off between the expected cumulative reward and the variance. Thus, while the goal of the risk-neutral MDP problem is to find the parameter θ that maximizes the total reward, we consider the mean-variance trade-off between the cumulative expected reward and the variance of the cumulative reward R in the MVRL problem. Note that even when the reward r is deterministic, the cumulative reward is a random variable owing to the stochastic policy, and there exists the mean-variance trade-off. In addition, even if the optimal policy is deterministic, the proposed method empirically has the potential to improves the stability of the training from the observations in the experiments of Section 5.3.

heading: 3 EXISTING MVRL METHODS

text: In this section, we introduce existing studies of MVRL.

heading: 3.1 CONSTRAINED TRAJECTORY-VARIANCE PROBLEM

text: Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) formulated MVRL by a constrained optimization problem defined as maxθ∈Θ Eπθ [R] s.t. Varπθ (R) ≤ η. In their formulation, the goal is to maximize the expected cumulative reward with controlling the trajectory-variance at a certain level. To solve this problem, Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) consider a penalized method defined as maxθ∈Θ Eπθ [R] − δg ( Varπθ (R) − η ) , where δ > 0 is a constant and g : R → R is a penalty function, such as g(x) = x or g(x) = x2.

heading: 3.1.1 DOUBLE SAMPLING ISSUE

text:
Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) report the double sampling issue in MVRL, which requires sampling from two different trajectories for estimating the policy gradient. For instance, in an episodic MDP with the discount factor γ = 1 and the stop-
ping time τ = min{t | St = S∗}, the gradients of Eπθ [R], Eπθ [ R2 ] , and ( Eπθ [R] )2 are given
as ∇θEπθ [R] = Eπθ [R ∑τ t=1 ∇θ log πθ(St, At)], ∇θEπθ [ R2 ] = E [ R2 ∑τ t=1 ∇θ log πθ(St, At) ] ,
and ∇θ ( Eπθ [R] )2 = 2Eπθ [R]∇θEπθ [R] (Tamar et al., 2012). Besides, the gradient of the vari-
ance is given as Eπθ [ R2 ∑τ t=1 ∇θ log πθ(St, At) ] − 2Eπθ [R]∇θEπθ [R]. Because optimizing the policy πθ using the gradients directly is computationally intractable, we replace them with their unbiased estimators. Suppose that there is a simulator generating a trajectory k with {(Skt , Akt , r(Skt , Akt ))}τ k t=1, where τ k is the stopping time of the trajectory. Then, we can naively
construct unbiased estimators of Eπθ [R] and Eπθ [ R2 ] as ∇̂θEπθ [R] = R̂k ∑τk t=1 ∇θ log πθ(Skt , Akt )
and ∇̂θEπθ [ R2 ] = ( R̂k )2 ∑τk t=1 ∇θ log πθ(Skt , Akt ), where R̂k is a sample approximation
of Eπθ [R] at the episode k. However, obtaining an unbiased estimator of ∇θ ( Eπθ [R] )2 = 2Eπθ [R]∇θEπθ [R] is difficult because it requires sampling from two different trajectories for approximating Eπθ [R] and ∇θEπθ [R]. This issue makes the optimization problem difficult.

heading: 3.1.2 EXISTING SOLUTIONS TO DOUBLE SAMPLING ISSUE

text:
For the double sampling issue, Tamar et al. (2012), Prashanth & Ghavamzadeh (2013), and Xie et al. (2018) presented the following solutions.
Multi-time-scale stochastic optimization: The proposed methods of Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) are based on stochastic approximation to find an equilibrium point of an ordinary differential equation and saddle point of the objective, respectively.
Coordinate descent optimization: Xie et al. (2018) proposed using the Legendre-Fenchel dual transformation with coordinate descent algorithm. First, based on Lagrangian relaxation, Xie et al. (2018) set an objective function as maxθ∈Θ Eπθ [R] − δ (Varπθ (R)− η). Then, Xie et al. (2018) transformed the objective function as maxθ∈Θ,y∈R 2y ( Eπθ [R] + 12δ ) −y2−Eπθ [ R2 ] and estimated a parameter by solving the optimization problem via a coordinate descent algorithm.
Weakness of existing approaches: The multi-time-scale approaches by Tamar et al. (2012) and Prashanth & Ghavamzadeh (2013) are known to be sensitive to the choice of the step-size schedules, which are not easy to be controlled (Xie et al., 2018). The method by Xie et al. (2018) does not reflect the constraint condition η as shown above; that is, in the objective function of Xie et al. (2018), there exits penalty coefficient δ, but does not exist the constraint condition η. Note that the problem of Xie et al. (2018) is owing to their objective function based on the penalty function g(x) = x: Eπθ [R]− δ (Varπθ (R)− η), in which the first derivative does not include η. In addition, when using quadratic function as g(x) = x2 to consider η, we cannot remove E[R2] even with Legendre-Fenchel dual; that is, the method also suffers the double sampling issue.

heading: 3.2 CONSTRAINED PER-STEP VARIANCE PROBLEM

text:
Bisi et al. (2020) and Zhang et al. (2020) proposed solving a constrained per-step variance problem for MVRL. Bisi et al. (2020) showed that the per-step variance Varπθ (R) ≤ Varπθ (r(St,At))
(1−γ)2 , which implies that the minimization of the per-step variance Var(r(St, At)) also minimizes trajectory-variance Varπθ (R). Therefore, they train a policy πθ by maximizing Eπθ [r(St, At)] − κVarπθ (r(St, At)), where κ > 0 is a parameter of the penalty function. The methods of Bisi et al. (2020) and Zhang et al. (2020) are based on the trust region policy optimization (Schulman et al., 2015) and coordinate descent with Legendre-Fenchel duality (Xie et al., 2018), respectively.

heading: 3.3 CONSTRAINED CUMULATIVE EXPECTED REWARD PROBLEM

text:
While existing MVRL studies mainly focus on a constrained trajectory-variance problem, a constrained cumulative expected reward optimization is also frequently used in practical situations, such as finance (Markowitz, 1952; Markowitz et al., 2000). In the constrained cumulative expected
reward problem, we solve the following problem:
min θ∈Θ Varπθ (R) s.t. Eπθ [R] = ξ. (1)
Our proposed EQUM framework is based on this motivation; that is, variance minimization. As shown in the following section, from a computational perspective, there is a critical difference between the constrained cumulative expected reward and trajectory-variance problems.

heading: 4 EQUM FRAMEWORK

text:
In this paper, as a novel approach for MVRL, we propose a Expected Quadratic Utility maximization RL (EQUM). In economic model, by using two parameters α > 0 and β > 0, the quadratic utility function for the cumulative return R is defined as U(R;α, β) = αR − 12βR
2 for α > 0, β ≥ 0 (Luenberger, 1998). The quadratic utility function captures the preference of a risk-averse agent over the cumulative return R. Let us consider the expected quadratic utility function defined as
Eπθ [U(R;α, β)] = αEπθ [R]− 1
2 βEπθ
[ R2 ] = αEπθ [R]− 1 2 β ( Eπθ [R] )2 − 1 2 βEπθ [( Eπθ [R]−R )2] . (2)
In the EQUM framework, we train a policy by maximizing the expected quadratic utility function.

heading: 4.1 INTERPRETATIONS

text:
Here, we introduce four interpretations of EQUM. We can interpret the EQUM as an approach for (i) an expected utility maximization, (ii) a targeting optimization problem to achieve an expected cumulative reward ζ, (iii) a constrained trajectory-reward problem with a quadratic penalized function, and (iv) an expected cumulative reward maximization with regularization.
First, we discuss the connection with respect to training an agent to achieve a predefined return (Berger, 1985). Let ζ be a target return that the algorithm aims to achieve. Then, we consider the mean squared error (MSE) minimization between the expected deviation of the return and ζ:
argmin θ∈Θ J(θ; ζ) = argmin θ∈Θ
Eπθ [( ζ −R )2] (3)
We can decompose the MSE into the bias and variance as follows: Eπθ [( ζ −R )2] = ( ζ − Eπθ [R]
)2︸ ︷︷ ︸ Bias +2Eπθ [( ζ − Eπθ [R] )( Eπθ [R]−R )] ︸ ︷︷ ︸
0
+Eπθ [( Eπθ [R]−R )2]︸ ︷︷ ︸ Variance
= ζ2 − 2ζEπθ [R] + ( Eπθ [R] )2 + Eπθ [( Eπθ [R]−R )2] .
Thus, the minimization problem (3) trains the policy πθ to consider the trade-off between the bias( ζ − Eπθ [R] )2 and variance Eπθ [( Eπθ [R]−R )2] (Section 9.5 of Luenberger (1998)). Moreover, we find that the EQUM is equivalent to the reward-targeting optimization when ζ = αβ ; that is,
argmin θ∈Θ J
( θ; α
β
) = argmin
θ∈Θ
( α
β
)2 − 2α
β Eπθ [R] +
( Eπθ [R] )2 + Eπθ [( Eπθ [R]−R )2] = argmax
θ∈Θ Eπθ [U(R;α, β)] .
Second, the bias-variance trade-off heuristically provides a solution to a constraint optimization problem (1) with the constraint ξ = ζ = αβ by solving the following penalized problem:
min θ∈Θ
Varπθ (R) + ( Eπθ [R]− ξ )2 . (4)
Third, we can regard the quadratic utility function as an expected cumulative reward maximization with a regularization term defined as E [ R2 ] ; that is, minimization of the risk R(πθ):
R(θ) = −Eπθ [R]︸ ︷︷ ︸ Risk of expected cumulative reward maximization
+ ψEπθ [ R2 ]︸ ︷︷ ︸ Regularization term
(5)
where ψ > 0 is a regulation parameter and ψ = β2α = 1 2ζ . As ζ → ∞, R(πθ) → −Eπθ [R].

heading: 4.2 MERITS OF THE EQUM FRAMEWORK

text:
In this section, we present two advantages of the EQUM framework. The first advantage is in computation. The EQUM framework is an MVRL method. However, compared with existing MVRL methods, which involve the double sampling issue, the computation of the EQUM framework is much simpler because we transform the MVRL problem (4) into the optimization problem without the term (Eπθ [R])
2 (see (2) and (5)). The second advantage is that it provides a variety of interpretations. Because the EQUM framework can interpret economic theory, it is applicable for modeling economic dynamics. In addition, as one of the MVRL methods, it is suitable for various real-world applications, such as finance (Deng et al., 2016) and playing games. By regarding the proposed EQUM as a regularization framework, we can combine it with existing RL methods.

heading: 4.3 IMPLEMENTATIONS OF THE EQUM FRAMEWORK

text:
Here, we discuss the implementations of the EQUM framework.
Simplest policy gradient with EQUM With the EQUM framework, we introduce a main algorithm with simplest policy gradient (SPG) algorithm (Brockman et al., 2016), which is also called the REINFORCE algorithm (Williams, 1992). For an episode k with length n, the method replaces the expectations Eπθ [ R ] and Eπθ [ R2 ] with the sample approximations ∑n t=1 γ
t−1r(St, At) and(∑n t=1 γ t−1r(St, At) )2 , respectively (Brockman et al., 2016). Then, the unbiased gradients are
∇̂θEπθ [R] = R̂k n∑ t=1 ∇θ log πθ(Skt , Akt ) and ∇̂θEπθ [ R2 ] = ( R̂k )2 n∑ t=1 ∇θ log πθ(Skt , Akt ).
Therefore, for a sample approximation R̂k of Eπθ [ R2 ] at the episode k, we optimize the policy with
ascending the following gradient: ( αR̂k − 12β ( R̂k )2)∑n t=1 ∇θ log πθ(Skt , Akt ).
Actor-critic with EQUM: For another combination with the EQUM framework, we apply an actor-critic (AC) based algorithms, which is also refereed to as the advantage actor-critic (A2C) algorithm (Williams & Peng, 1991; Mnih et al., 2016). Extending the AC algorithm, for an episode k with the length n, we train the policy by a gradient defined as
∇θ log πθ(Skt , Akt ) {( αR̃kt:t+n−1 − 1
2 βR̃k,2t:t+n−1
) − ( αM (1)
ω̂ (1) k
(Skt )− 1
2 βM
(2) ω̂ (2) k (Skt )
)} ,
where R̃kt:t+n−1 = R̂ k t:t+n−1+γ nM (1)
ω̂ (1) k
(Skt+n), and R̃ k,2 t:t+n−1 = ( R̂kt:t+n−1 + γ nM (1)
ω̂ (1) k
(Skt+n)
)2 ,
and M (1) ω̂
(1) k
(Skt ) and M (2)
ω̂ (2) k
(Skt ) are value functions approximating E[Rt+1:∞] and E[R2t+1:∞] with
parameters ω̂(1)k and ω̂ (2) k , respectively.
For other RL algorithms, we can heuristically extend our proposed framework EQUM to accept the other RL algorithms by adding E[R2] as regularization term.
Determining α and β: Next, we discuss the parameter tuning of α and β, which are equivalent to ζ, ξ, and ψ). As explained, the parameter ψ has several meanings based on the interpretations of the EQUM, such as the quadratic utility function, targeting optimization, and constrained optimization. In addition, from the regularization perspective, we can adjust the parameter
to maximize the expected cumulative reward in the validation data. Thus, we propose the following four directions for the parameter tuning. First, in economic applications, such as finance, we choose β2α = ψ based on the theoretical economic assumptions and economic empirical studies (Ziemba et al., 1974; Kallberg & Ziemba, 1983) (Appendix A). For instance, Capital Asset Pricing Model (CAPM), which is one of the most popular economic models, is also base on the quadratic utility function (Sharpe, 1964; Lintner, 1965; Mossin, 1966). Second, we set ζ = 12ψ as the targeted reward to achieve. Third, we regard the parameter ψ as the constrained problem (1). Fourth, through cross-validation, we optimize the regularization parameter ψ.

heading: 5 EXPERIMENTS

text: This section investigates the empirical performance of the proposed EQUM with synthetic and realworld financial datasets. The goal of these experiments is to construct a portfolio with a wellcontrolled mean and variance through algorithms. In addition, we also show experimental results using CartPole and Atari games. Note that the per-step rewards of portfolio selection are stochastic variables and those of CartPole and Atari games are deterministic variables. However, even when the reward is deterministic, the cumulative reward has randomness owing to the stochastic policy.

heading: 5.1 EXPERIMENTS WITH SYNTHETIC PORTFOLIO SELECTION DATASETS

text:
Following Tamar et al. (2012) and Xie et al. (2018), we artificially generate a portfolio dataset. Let us consider a portfolio composed of two types of assets: a liquid asset, which has a fixed interest rate rl = 1.001, and a non-liquid asset, which has a time-dependent interest rate rnl(t) ∈ {1.1, 2}. While we can sell the liquid asset at every time step t = 1, 2, . . . , T , we can sell the non-liquid asset only after a maturity period of N steps; that is, when holding 1 liquid asset, we obtain 1.001 per period; when holding 1 non-liquid asset at the t-th period, we obtain 1.1 or 2 at the t+N -th periods. In addition, the non-liquid asset has some risk of not being paid with a probability prisk; that is, if the non-liquid asset defaulted during the N periods, we cannot obtain any profits by having the asset. In this setup, a typical investment strategy is to construct a portfolio using both liquid and non-liquid assets to control the mean and variance. In our model, the investor may change his portfolio by investing a fixed fraction α = 0.2 to the non-liquid asset at each time step. As a performance metric of the portfolio, we focus on the mean and variance of the cumulative reward when having the cash 1 at the first period and investing the cash for 50 periods following an algorithm.
In particular, we aim to investigate the sensitivity of the EQUM against the parameter ψ = 2/ζ and compare the EQUM with the standard SPG and AC algorithms. We apply the SPG algorithm with the EQUM (Section 4.3) to the synthetic datasets. For ζ, we use ζ = 1, 2, 3, 4, 5, 6. Note that from the regularization perspective with a parameter ψ, EQUM with ψ = 0.1 is equal to minimize the MSE between the cumulative reward and ζ = 5 = 1/(2 × 0.1). We train the model with 500 episodes. In Figure 1, we calculate the average reward and standard deviation at each episode of the training process over 1, 000 trials. In Table 1, using the trained model for the test environment, we show the average reward (AR) and standard deviation (SD) by conducting 1, 000 trials.
For each trial, we compute the AR and SD over 100 trial and compute the average of each AR and SD over 1, 000 trials. From Figure 1 and Table 1, we can clearly confirm that the EQUM framework
realize the control of the mean-variance trade-off. As shown in the results, we can reduce the variance by increasing ζ. The MSE from target ζ is shown in Table 2. In Figure 2, for the EQUM with various ζ and the method of Tamar et al. (2012) (Tamar) with various variance constraints (var), we plot the AR and variance (Var) computed on the train environment over 1, 000 trials. We show the Var instead of the SD because the Tamar controls the variance. Compared with Tamar, the EQUM returns Pareto efficient portfolios; that is, higher AR and lower Var. We consider unlike the direct optimization of the EQUM, the Tamar suffers from the complicated optimization mechanism. Figure 3 in Appendix B.1 corresponds to the test environment version of Figure 2.
In Appendix B.2, we also show experimental results using another synthetic dataset of Americanstyle option following Tamar et al. (2014) and Xie et al. (2018).

heading: 5.2 EXPERIMENTS WITH REAL-WORLD DATASET

text:
We use well-known benchmarks called Fama & French (1992) (FF) datasets to ensure the reproducibility of the experiment. We use the FF25, FF48 and FF100 datasets. The FF25 and FF 100 dataset includes 25 and 100 portfolios formed based on size and book-to-market ratio; the FF48 dataset contains 48 portfolios representing different industrial sectors. We formulate the problem by an episodic MDP. We use all datasets covering monthly data from July 1980 to June 2020. The state is past 12 months returns of each asset and the action is defined as portfolio weight; that is, the number of actions is equal to that of assets. The reward is obtained as the portfolio return. Here, the portfolio return at time 1 ≤ t ≤ T is defined as yt = ∑m j=1 yj,twj,t−1, where yj,t is the return of j asset at time t, wj,t−1 is the weight of j asset in the portfolio at time t − 1, and m is the number of assets. The length of the episode is 1 years (12 months). For the stochastic policy, we adopt a three-layer feed-forward neural network with the ReLU activation function where the number of units in each respective layer is equal to the number of assets, 100, 50, and the number of actions. We use the softmax function for the output layer.
Portfolio models: We use the following portfolio models. An equally-weighted portfolio (EW) weights the financial assets equally (DeMiguel et al., 2007). A mean-variance portfolio (MV) computes the optimal variance under a mean constraint (Markowitz, 1952). For computing the mean vector and covariance matrix, we use the latest 10 years (120 months) data. An Kelly growth optimal portfolio with ensemble learning (EGO) is proposed by Shen et al. (2019). We set the number of resamples as m1 = 50, the size of each resample m2 = 5τ , the number of periods of return data τ = 60, the number of resampled subsetsm3 = 50, and the size of each subsetm4 = n0.7, wherem is number of assets; that is, m = 25 in FF25, m = 48 in FF48 and m = 100 in FF100. A portfolio blending via Thompson sampling (BLD) is proposed by Shen & Wang (2016). We use the latest 10 years (120 months) data to compute for the sample covariance matrix and blending parameters. A policy gradients with variance related risk criteria (Tamar) is proposed by Tamar et al. (2012). We set the target variance terms η as 250,500,1000. A block coordinate ascent algorithm proposed by Xie et al. (2018), which is referred to as Mean-Variance Portfolio (MVP). We set the regularization parameters δ as 10,100,1000. Then, let us denote the SPG with the EQUM framework as EQUM. The parameter ψ is chosen from 1/3, 2/3, and 1. For optimizing Tamar, MVP and EQUM, we set the Adam optimizer with learning rate 0.01 and weight decay parameter 0.1. We train the neural networks for 10 episodes. Each portfolio is updated by sliding one-month-ahead.
Performance metrics: The following measures widely used in finance to evaluate portfolio strategies (Brandt, 2010) are chosen. The cumulative return (CR), annualized risk as the standard de-
MaxDD↓ 0.55 0.72 0.58 0.36 0.39 0.49 0.38 0.52 0.65 0.46 0.24 0.31 0.39 viation of return (RISK) and risk-adjusted return (R/R) are defined as follows: CR = ∑T t=1 yt,
RISK = √
12 T−1 ∑T t=1(yt − CR/T )2, and R/R = 12 T ×CR/RISK. R/R is the most important mea-
sure for a portfolio strategy. We also evaluate the maximum draw-down (MaxDD), which is another widely used risk measure Magdon-Ismail & Atiya (2004) for the portfolio strategy. In particular, MaxDD is the largest drop from a peak defined as MaxDD = mint∈[1,T ] ( 0, Wtmaxτ∈[1,t]Wτ − 1 ) ,
where Wk is the cumulative return of the portfolio until time k; that is, Wt = ∏t t′=1(1 + yt′).
Table 3 reports the performances of the portfolios. In almost all cases, the EQUM portfolio achieves the highest R/R and the lowest MaxDD. Therefore, we can confirm that the EQUM portfolio has a high R/R, and avoids a large drawdown. The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP, and EQUM is shown in Appendix B.3. Except for FF48’s MVP, the objective itself is smaller than EQUM’s. Since the values of the objective is the same as the RR, we can empirically confirm that the better optimization, the better performance.

heading: 5.3 EXPERIMENTS WITH CARTPOLE AND ATARI GAMES

text: We also conduct experiments using CartPole and Atari games, where the reward is given as deterministic, and the randomness of the cumulative reward depends only on the stochastic policy. We investigate the sensitivity of ψ = 1/(2ζ) with CartPole and the compared the performance of the EQUM framework with that of Tamar et al. (2012) and Xie et al. (2018). The results are shown in Appendix B.4. In many experimental results, contrary to our expectations, we observed that the EQUM also improves the expected cumulative reward, not only the variance. We hypothesize that this is because there is often a limit on the cumulative rewards achieved by the standard expected cumulative reward maximizing algorithms. For instance, when the reward at each period is 1, and the discount factor is 0.99, the infinite sum is 100. In such a case, instead of naive reward maximization, MSE minimization against the target reward 100 may result in a more stable performance empirically. We also hypothesize that even if an optimal policy is deterministic, the EQUM can improve the stability of the training process by reducing the variance. This observation is an open problem related to exploration and exploitation trade-off. Therefore, unless we could increase the cumulative reward infinitely, the proposed EQUM framework can stabilize the performance. From this aspect, we can confirm that the EQUM provides a regularization effect.

heading: 6 CONCLUSION

text: In this paper, we proposed an EQUM framework as a variant of MVRL. Compared with existing MVRL methods, the EQUM framework is computationally friendly. The proposed EQUM framework also includes various interpretations, such as targeting optimization and regularization and is suitable for many real-world applications, such as finance and playing games. We investigated the effectiveness of the EQUM framework compared with the standard RL and existing MVRL methods through the experiments. In the results, the proposed method successfully controls the mean-variance trade-off. As an open problem, we also observed that even when an optimal policy is deterministic, the proposed method improves performance. We hypothesize that the proposed method affects the stabilization of the training process.

heading: A ECONOMIC THEORY AND QUADRATIC UTILITY FUNCTION

text:
Considering the mean-variance trade-off in a portfolio and economic activity is an essential task in economics as Tamar et al. (2012) and Xie et al. (2018) pointed out. The mean-variance trade-off is justified by assuming either quadratic utility function to the economic agent or multivariate normal distribution to the financial assets. By assuming the quadratic utility function or the normal distribution, we can assume that the expected utility function of the agent is maximized by maximizing the expected reward and minimizing the variance. Based on this observation, Markowitz (1952) proposed the following steps for providing a portfolio to an economic agent (Also see Markowitz (1959), page 288, and Luenberger (1998)):
• Constructing portfolios minimizing the variance under several reward constraint; • Among the portfolios constructed in the first step, the economic agent chooses a portfolio
maximizing the utility function.
Therefore, the goal of Markowitz’s portfolio is not only to construct the portfolio itself but also to maximize the expected utility function of the agent. In traditional economics, this two-step is adopted because directly predicting the reward and variance to maximize the expected utility function is difficult; therefore, first gathering information based on analyses of an economist, then we construct the portfolio using the information and provide the set of the portfolios to an economic agent. However, owing to the recent development of machine learning, we can directly represent the complicated economic dynamics using flexible models, such as deep neural networks. In addition, as Tamar et al. (2012) and Xie et al. (2018) reported, when constructing the mean-variance portfolio in RL, we suffer the double sampling issue. Therefore, in this paper, we aim to achieve the original goal of the mean-variance approach; that is, the expected utility maximization. Note that this idea is not restricted to financial applications but can be applied to applications where the agent utility can be represented only by the mean and variance. In the following subsection, we review the existing studies of finance and quadratic utility function.
A.1 MARKOWITZ’S PORTFOLIO AND CAPITAL ASSET PRICING MODEL
Markowitz’s portfolio is known as the mean-variance portfolio (Markowitz, 1952; Markowitz et al., 2000). Constructing the mean-variance portfolio is motivated by the agent’s expected utility maximization. When the utility function is given as the quadratic utility function, or the financial asset follows the multivariate normal distribution, a portfolio maximizing the agent’s expected utility function is given as a portfolio with minimum variance under a certain standard expected reward.
The Capital Asset Pricing Model (CAPM) theory is a concept which is closely related to Markowitz’s portfolio (Sharpe, 1964; Mossin, 1966; Lintner, 1965). This theory theoretically explains the expected return of investors when the investor invests in a financial asset; that is, it derives the optimal price of the financial asset. To derive this theory, as well as Markowitz’s portfolio, we assume the quadratic utility function to the investors or the multivariate normal distribution to the financial assets.
Merton (1969) extended the static portfolio selection problem to a dynamic case. Fishburn & Porter (1976) studied the sensitivity of the portfolio proportion when the safe and risky asset distributions change under the quadratic utility function. Thus, there are various studies investigating relationship between the utility function and risk-averse optimization (Tobin, 1958; Kroll et al., 1984; Bulmuş & Özekici, 2014; Bodnar et al., 2015).
A.2 EMPIRICAL STUDIES ON THE UTILITY FUNCTIONS
The standard financial theory is built on the assumption that the economic agent has the quadratic utility function. For supporting this theory, there are several empirical studies to estimate the parameters of the quadratic utility function. Ziemba et al. (1974) investigated the change of the portfolio proportion when the parameter of the quadratic utility function changes using the Canadian financial dataset. Recently, Bodnar et al. (2018) investigate the risk parameter (α and β in our formulation of the quadratic utility function) using the markets indexes in the world. They found that the utility function parameter depends on the market data model.
A.3 CRITICISM
For the simple form of the quadratic utility function, the financial models based on the utility are widely accepted in practice. However, there is also criticism that the simple form cannot capture the real-world complicated utility function. For instance, Kallberg & Ziemba (1983) criticized the use of the quadratic utility function and proposed using a utility function, including higher moments. This study also provided empirical studies using U.S. financial dataset for investigating the properties of the alternative utility functions. However, to the best of our knowledge, financial practitioners still use financial models based on the quadratic utility function. We consider this is because the simple form gains the interpretability of the financial models.

heading: B DETAILS OF EXPERIMENTS

text:
In this section, we show the additional experiments and describe the details of the experiments. To implement SPG and AC algorithms, we mainly follow the Pytorch example1.
B.1 PARETO EFFICIENCY ON THE TEST ENVIRONMENT OF THE SYNTHETIC PORTFOLIO SELECTION DATASET
For investigating the Pareto efficiency, we show the AR and Var of the train environment in Figure 2 of Section 5.1. Here, we also show the AR and Var of the test environment in Figure 3.
B.2 EXPERIMENTS USING THE SYNTHETIC AMERICAN-STYLE OPTION DATASET
Among various options in finance, an American-style option refers to a contract that we can execute an option right at any time before the maturity time τ ; that is, a buyer who bought a call option has a right to buy the asset with the call option strike price Wcall at any time; a buyer who bought a put option has a right to sell the with the call option strike price Wput at any time.
In the setting of Tamar et al. (2014) and Xie et al. (2018), the buyer simultaneously buy call and put options, which have the strike price Wcall = and Wput =, respectively. The maturity time is set as τ =. If the buyer executes the option at time t, the buyer obtains a reward rt = max(0,Wput − xt)+max(0, xt−Wcall), where xt is an asset price. We set x0 = 1 and define the stochastic process as follows: xt = xt−1fu with probability 1 and xt = xt−1fd with probability 1, where fu and fd. These parameters follows Xie et al. (2018). Under this setting, we investigate the performance of our EQUM. For the EQUM, we use ζ = 0.1, 0.3, 0.5, 0.7, 1, 1.3. For the other settings, we follow the previous experiment for portfolio management.
1https://github.com/pytorch/examples/tree/master/reinforcement_learning
In Figure 4, we calculate the average reward and standard deviation at each episode of the training process by conducting 1, 000 trials. In Table 4, using the trained model for the test environment, we show the average reward (AR) and standard deviation (SD) by conducting 1, 000 trials. We show the MSE between the realized reward and the target reward ζ in Table 5, where the lowest MSE method is highlighted in bold. As well as the experimental results with the synthetic portfolio dataset, we can confirm that the EQUM can control the risk well. In addition, as well as the experiments with CartPole and Atari games, we can observe that the EQUM also increases the expected reward contrary to our expectation.
B.3 DETAILS OF EXPERIMENTS OF PORTFOLIO OPTIMIZATION
The real objective (minimizing variance with a penalty on return targeting) for Tamar, MVP and EQUM is shown in Table 6. Except for FF48’s MVP, the objective itself is smaller than EQUM’s. Since the values of objective is the same as the RR, we can empirically confirm that the better optimization, the better performance.
We also divide the performance period into two for robustness checks. Table 7 shows the first-half results from July 2000 to June 2010 and the second-half results from July 2010 to June 2020. In almost all cases, the EQUM portfolio achieves the highest R/R.
B.4 DETAILS OF EXPERIMENTS WITH CARPOLE AND ATARI GAMES
In this section, we report the experimental performances of the proposed EQUM framework with well-known benchmarks. We investigate how the behaviors of existing RL methods are changed by adding the additional E[R2] term. We use a simple two-layer perceptron for modeling the policy following the Pytorch example (Paszke et al., 2019). For the SPG-based algorithms, we define the cumulative reward as finite sum with γ = 1 following (Tamar et al., 2012). For the AC-based algorithms, we define the cumulative reward as infinite sum with γ = 0.99 following (Prashanth & Ghavamzadeh, 2013). For the SPG-based algorithms, we define the cumulative reward as finite sum with γ = 1 following (Tamar et al., 2012). For the AC-based algorithms, we define the cumulative reward as infinite sum with γ = 0.99 following (Prashanth & Ghavamzadeh, 2013).
B.4.1 SENSITIVITY ANALYSIS ON ψ
First, we investigate the sensitivity of the EQUM framework to the parameter ψ using the CartPole. We apply the SPG and AC methods (Section 4.3) with the EQUM framework to the Cartpole problem, which is a classic control problem. We use ψ = 0.001, 0.002, 0.003, 0.005, 0.01, 0.1 and compare the results with the standard SPG and AC methods. For instance, from the targeting optimization perspective, EQUM with ψ = 0.001 is equal to minimize the MSE between the cumulative reward and ζ = 500 = 1/(2 ∗ 0.001). We train the model by 300 episodes. We calculate the average reward and standard deviation at each period by conducting 300 trials. The results are shown
in Figure 5. In the SPG results, we can confirm the mean-variance trade-off. In contrast, in the SPG with ψ = 0.001 and all AC results, the EQUM framework improves the expected cumulative reward. These results are contrary to our expectations because MVRL methods typically decrease the expected cumulative return to decrease the variance. We discuss this topic in more detail in Section 5.3.
B.4.2 EXPERIMENTS OF ATARI GAMES
When playing games, we often consider risk-control while maintaining a certain level of reward. Under this motivation, we benchmark the proposed EQUM framework on four Atari game tasks from the OpenAI gym (Brockman et al., 2016). Among the games, we choose BeamRider, Seaquest, Qbert, and SpaceInvaders in which SPG and AC methods work well. We use simplified environments in which the observations are the RAM of the Atari machine, consisting of only 128 bytes. We compare our methods with the standard SPG, and MVLR methods of Tamar et al. (2012) and Xie et al. (2018). We calculate the average reward (AR) at the last episode over 5 trials and the standard deviation (SD). In Table 8, we show the results of the SPG algorithm with the standard setting and EQUM framework, where ψ is chosen from 0.001, 0.003, 0.005, 0.01, 0.03, and 0.3. Note that when using ψ = 0.005, it is equal to minimize the MSE between the expected cumulative reward and the target 100. In almost all cases, the EQUM framework shows the better AR than the standard methods. As well as the previous sensitivity experiment, this result is contrary to our expectation. Among the games, we choose simplified BeamRider, Seaquest, Qbert, and SpaceInvaders in which the SPG algorithm work well, and the observations are the RAM of the Atari machine, consisting of only 128 bytes. We compare our methods with the standard SPG algorithms, and MVRL methods of Tamar et al. (2012) and Xie et al. (2018). We denote the SPG-based methods proposed by Tamar et al. (2012) and Xie et al. (2018) as Tamar and MVP, respectively. We calculate the average reward (AR) at the last episode over 5 trials and the standard deviation (SD). We choose the parameter of Tamar from 200 and 2000, which is denoted as b in Tamar et al. (2012). We choose the parameter of MVP from 1 and 10, which is denoted as λ in Tamar et al. (2012). The parameters are denoted as param in Table 8.