Actor-Critic Algorithms for Risk-Sensitive MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance-related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms for estimating the gradient and updating the policy parameters in the ascent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.
The authors present actor-critic algorithms for 
      learning risk-sensitive policies. They present infinite-horizon methods 
      for both the discounted reward and average reward cases, and present 
      empirical results demonstrating their use on a simulated traffic control 
      problem. -Quality: The paper appears to be sound and thorough. It 
      addresses an important problem. -Clarity: The paper is clear; I have 
      suggestions below. -Originality: To the best of my knowledge, the 
      approach and results are novel. -Significance: The paper improves our 
      understanding and ability to deal with risk in RL, which is important for 
      many problems. ----Specific suggestions and questions: 
      -"variance related" ... "variance-related" -Don't use 
      parenthetical references as nouns. ("similar to [21]...") -"...lacks 
      the monotonicity property..." I think a cite here or an explanation of 
      what "monotonicity" means in this setting is needed. -"...in order to 
      compute \nabla\Lambda^\theta(x^0), it would be enough to calculate 
      \nabla{}U^\theta(x^0)." I found this confusing; I think you mean because 
      we need to compute \nabla{}V anyway? I think a more clear statement about 
      how to go beyond "traditional" actor-critic we need only this additional 
      term would help, perhaps. -Lemma 1 does not fit into the column width. 
      -\kappa_1 is used before definition; mention what space \theta lives 
      in first in order for things to make sense. -In (5), \theta^i should 
      be \theta^{(i)} -In the SPSA case, since \Delta^{(i)} are Rademacher, 
      they could equally be written in the numerator. I understand this is not 
      quite "in the spirit" of the derivative approximation, where the step-size 
      would appear in the denominator, but I think the win of being able to 
      re-write (4) and (5) in exactly the same way makes it worth it. Then you 
      don't have to repeat yourselves later on, which would free up some space. 
      -Do you care what norm the projection operator is in? It would be 
      worth mentioning either way. -I think there is a missed 
      opportunity to discuss why the discounted reward case or the average 
      reward case is more appropriate for a particular problem. In the 
      average-reward case, the variance of R(s,a)---that is, the variance of the 
      reward random variable at a fixed state-action pair---does not impact the 
      objective, whereas in the discounted case it does. This seems like it 
      might matter. Can you comment on why one might prefer one over the other? 
      -Related, rather than "...it is the frequency of occurrence of 
      state-action pairs that determine the variability in the average reward." 
      I might say it is *only* the variability in frequency of occurrence. 
      -I think \cdot (or nothing at all) would be better than * in (19) 
      -"...is not significantly low..." I would say "poor," since lower is 
      better in this case; furthermore avoid the word significant unless you 
      will explain that you mean statistical significance, practical 
      significance, or both. -Give plots in Figure 2 the same x-range. 
      -RS-SF looks like it dominates RS in Figure 2b. Does it? Why might 
      this be? ADDENDUM: Is this reward or cost? If larger quantities are not 
      better, *do not call it reward*. Either negate it or change terminology. 
      -Assuming it is "cost" rather than reward, it looks like SF 
      returns might be stochastically smaller than RS-SF, even though it has 
      larger variance. In this case, why should the RS-SF policy be desirable? 
      Is it? In general, the argument that the reduction in variance is "worth 
      it" is not very convincing. Perhaps explain what the units mean and that 
      the difference is not really relevant. -"difficultly" -- 
      Spellcheck your submission. -Hyphenate before -based. 
      "perturbation-based, and so on."