Algorithms for CVaR Optimization in MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively new risk measure that addresses some of the shortcomings of the well-known variance-related risk measures, and because of its computational efficiencies has gained popularity in finance and operations research. In this paper, we consider the mean-CVaR optimization problem in MDPs. We first derive a formula for computing the gradient of this risk-sensitive objective function. We then devise policy gradient and actor-critic algorithms that each uses a specific method to estimate this gradient and updates the policy parameters in the descent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in an optimal stopping problem.
This paper derives policy gradient algorithms for risk-sensitive MDPs for the particular criterion CVaR - a recent and popular criterion. First, the author derive gradients for the objective based on a Lagrangian relaxation of the constrained optimization. This naturally turns into a policy gradient algorithm where the expected return that appears in the gradient is estimated from full trajectories (reinforce-like). They then propose a scheme to obtain incremental actor-critic versions, where the critic computes the value (and other quantities) of an augmented MDP convenient for gradient estimation. An experimental section illustrates these algorithms on a small problem.The paper is well-written, and the introduction clearly motivates the paper. The paper is quite technical, but I didn't have trouble understanding the notation and following the equations/algorithms. I am only mildly familiar with the literature on risk-sensitive MDPs so I can't provide insightful comments on the impact/significance of the contributions in the paper. My understanding is that, being incremental and compatible w/ function approximation, the algorithms in the paper have the potential to scale to larger domains than what is currently the norm with risk-sensitive planning.A couple of questions:- About the dependence of the policy on the accumulated cost and time, I understood that by making the feasibility assumption then you can restrict yourself to state-only (but stochastic) policies. Is that correct? Does it mean there might be better history-based policies that satisfy the CVaR constraint but that you are not considering? It might be worth clarifying this in the paper.[EDIT: Thanks for clarifying]- About the optimized policy, does converging to a saddle point in the Lagrangian necessarily imply that we reach any kind of local optimum in the original optimization problem? If not, then is it really fair to say that the algorithms converge to "locallly risk-sensitive optimal policies" (abstract)?   - In the intro, it is claimed that [27] (the closest competitor it seems) does not have a convergence proof for their policy-gradient algorithm, but a quick look at [27] reveals the following claim: "We [...] prove the convergence of the policy gradient algorithm to a local optimum". Maybe this is worth clarifying?[EDIT: Fair enough, I didn't pay attention to the arxiv version/date of submission.]While I appreciate the theoretical contribution, I was a bit disappointed by the experimental section, which does not really convey the impression of scalability and robustness (in contrast, [27] had experiments with the tetris domain using CVaR). Future work #3 seem particularly critical for sample-based algorithms in this risk setting, so it is a pity that this wasn't addressed in this work.Overall, a nice paper with clear contributions. Experiments are lacking but there is a lot of theoretical work.Some minor typos:- Equation 4: max lambda should have a non-negativity constraint(?)- line 1313:  xi_3 --> xi_4- line 269 (footnote): SPSA not yet defined. Note: I haven't verified all the proofs in the long supplementary material provided with the paper.