Projected Natural Actor-Critic
Natural actor-critics form a popular class of policy search algorithms for finding locally optimal policies for Markov decision processes. In this paper we address a drawback of natural actor-critics that limits their real-world applicability-their lack of safety guarantees. We present a principled algorithm for performing natural gradient descent over a constrained domain. In the context of reinforcement learning, this allows for natural actor-critic algorithms that are guaranteed to remain within a known safe region of policy space. While deriving our class of constrained natural actor-critic algorithms, which we call Projected Natural ActorCritics (PNACs), we also elucidate the relationship between natural gradient descent and mirror descent.
Summary The paper is about the proposal of a 
      class of constrained natural actor critics, where, for safety reasons, 
      policy parameters must remain in a subregion. The idea is to apply 
      natural actor critic algorithms, that update policy parameters by 
      following the estimated direction of the natural policy gradient and, 
      whenever the policy parameters get out of the safe region, the parameters 
      are projected back to allowed values. The authors show that natural 
      gradient ascent is a particular case of mirror ascent, and, being the 
      latter a constrained optimization algorithm, the projection can be simply 
      (and effectively) obtained by adding constraints to the policy parameters 
      values. Besides theoretically proving that the resulting projection is 
      compatible with the natural policy gradient, a simple example and two more 
      complex case studies have been introduced to evaluate the performance of 
      the proposed solution and the negative effects that can derive in critical 
      systems when either unconstrained optimization or a wrong projection 
      method are used. Quality The paper is technically sound. The 
      theoretical claims are correctly proved and the effectiveness of the 
      proposed approach is empirically evaluated in medium-complexity domains. 
      The authors have made a good work in highlighting the advantages of 
      the proposed approach, while they have not mentioned to potential weakness 
      and no future research direction has been proposed. For instance, I 
      would like to read some discussion about the complexity of computing the 
      projection and to see some example where the constraints over the 
      parameters are not linear. Clarity The paper is well written. 
      The paper is structured into many sections. I suggest to reorganize the 
      paper into a smaller number of sections that collects the current sections 
      as subsection. For instance, I would group sections 3,4,5, sections 
      6,7, sections 8,9, and section 10,11. Furthermore, I would move the 
      related work section at the end, before the conclusions. I found 
      particularly instructing the example of Section 7, that allows the reader 
      to clearly visualize how the proposed approach moves in the parameter 
      space and what may happen by following other (naive) approaches. For 
      what concerns the reproducibility of experimental results, the example in 
      Section 7 can be easily reproduced, while for the two case studies some 
      details are missing about the domain. In general, no information have 
      been reported about the value of the step size used to update the 
      parameters along the gradient direction. Originality As far as 
      I know, the proposed approach is novel. In the related work section, the 
      authors cite other approaches to problems similar to the one faced in this 
      paper. As the authors state in the conclusions, the contribution is a 
      "simple" modification to standard natural actor-critic algorithms to 
      include projection. Such modification is simple since, by choosing the 
      correct proximal function, natural gradient ascent can be realized through 
      the constrained optimization carried out by mirror descent. 
      Significance The setting faced by this paper is really 
      relevant. In the last years, policy search and, in particular, policy 
      gradient approaches have been largely used to learn policy for complex 
      continuous tasks. On the other hand, perform an unconstrained 
      optimization in the policy space may be dangerous. This paper advances 
      the state of the art by introducing a principled way to constrain the 
      search for one prominent class of policy gradient algorithms: natural 
      actor critic algorithms. I think that this line of research is 
      interesting for many researchers (especially in the field of robotics) and 
      this paper can have a positive impact on future researches.