Policy Gradient for Coherent Risk Measures
Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific risk-measures, such as the variance or conditional value at risk (CVaR). In this work, we extend the policy gradient method to the whole class of coherent risk measures, which is widely accepted in finance and operations research, among other fields. We consider both static and time-consistent dynamic risk measures. For static risk measures, our approach is in the spirit of policy gradient algorithms and combines a standard sampling approach with convex programming. For dynamic risk measures, our approach is actor-critic style and involves explicit approximation of value function. Most importantly, our contribution presents a unified approach to risk-sensitive reinforcement learning that generalizes and extends previous results.
The paper proposes a new formula for the class of static coherent risks from which policy gradients for many common risk measures can be derived analytically. It also presents a general sampling-based algorithm for cases where no analytical solution is available and shows consistency guarantees of the policy gradient estimates. Eventually, the paper provides expressions for policy gradients of Markov coherent dynamic risk measures.Significance and quality:The unified derivation of policy gradients for coherent risk measures is intriguing and a valuable and novel contribution to the risk-sensitive decision making literature. Nevertheless, analytical solutions of the Lagrangian saddle-points are only available for some risk measures.The proposed sample-based approximation scheme could allow us to apply the derived results for general coherent risk measures by avoiding to derive analytical forms of the saddle points. However, the paper does not provide any empirical evidence that the proposed estimation scheme is viable in practice. An experimental comparison between the using the sample-based policy gradients and the analytical ones in different problems, e.g. for CVaR would be extremely helpful.   Originality: To the best of my knowledge, the derived policy gradients are - in this general form - are novel.Clarity: The paper is generally well written, however, it appears to me that the clarity of the main paper is hindered considerably by number of different issues attempted to be addressed in the eight pages. Especially the dynamic risk policy gradients in Section 5 are presented on a very high level and the lack of any algorithm details and experimental evaluations do not allow the reader to judge whether the derived policy gradients yield approaches that are viable in practice. It seems to me this content is rather suited for a longer journal publication. An alternative that I would see as beneficial is to split this paper and rather focus on static risk measures which allows to make the paper more standalone and provide more details and a more comprehensive empirical evaluation beyond the very basic toy example in the current submission.