Adaptive Step-Size for Policy Gradient Methods
In the last decade, policy gradient methods have significantly grown in popularity in the reinforcement-learning field. In particular, they have been largely employed in motor control and robotic applications, thanks to their ability to cope with continuous state and action domains and partial observable problems. Policy gradient researches have been mainly focused on the identification of effective gradient directions and the proposal of efficient estimation algorithms. Nonetheless, the performance of policy gradient methods is determined not only by the gradient direction, since convergence properties are strongly influenced by the choice of the step size: small values imply slow convergence rate, while large values may lead to oscillations or even divergence of the policy parameters.
Step-size value is usually chosen by hand tuning and still little attention has been paid to its automatic selection. In this paper, we propose to determine the learning rate by maximizing a lower bound to the expected performance gain. Focusing on Gaussian policies, we derive a lower bound that is second-order polynomial of the step size, and we show how a simplified version of such lower bound can be maximized when the gradient is estimated from trajectory samples. The properties of the proposed approach are empirically evaluated in a linear-quadratic regulator problem.
This paper proposes a new policy-gradient 
      reinforcement learning method in which the step size is set adaptively by 
      maximizing a lower bound on the expected performance gain. The authors 
      first consider the case where the policy gradient is known and establish a 
      lower bound on the difference in performance between the original policy 
      and the modified policy resulting from an update that follows the policy 
      gradient. Since the above bound is a fourth-order polynomial, the authors 
      then consider the special case in which the stochastic policy is 
      represented as a Gaussian distribution with a fixed standard deviation and 
      a mean that is a linear combination of the state features. In this case, 
      they derive a lower bound that is quadratic in the step size and has a 
      single maximum for positive step sizes. Next, they remove the assumption 
      that the policy gradient is known and establish a high-probability lower 
      bound given an epsilon-optimal estimate of the policy gradient. Finally, 
      they show how two existing techniques, REINFORCE and G{PO)MDP/PGT, can 
      provide such policy gradient estimates, leading to an adaptive step size 
      policy gradient method that does not require a model of the environment 
      and can learn only from sample trajectories. The proposed method is 
      evaluated in a simple simulation experiment using a 1-dimensional toy 
      problem. Overall, this is a high-quality, well written paper. The 
      authors make a good argument that most research on policy-gradient methods 
      has focused on finding better gradient estimators and thus there remains 
      room for improvement in automating the step size in such algorithms. The 
      theoretical results in the paper make a substantial contribution by 
      providing a principled framework for the automatic selection of such step 
      sizes by maximizing lower bounds on expected performance gain. The paper 
      contributes not only a specific algorithm but more general results that 
      could be used to derive algorithms with different policy representations 
      and gradient estimators. The main weakness in the paper is the 
      empirical evaluation, which considers only a toy problem involving the 
      optimization of one policy parameter. Results obtained using the true 
      gradient show that manual choices of the step size can lead to slow 
      learning (too small) or divergence (too big) whereas the adaptive method 
      always converges within the threshold. For large values of the standard 
      deviation of the Gaussian policy, the adaptive method performs better than 
      any of the fixed step-sizes tested. However, for small standard deviations 
      it performs worse, which the authors suggest reflects an inherent 
      trade-off between the determinism of the policy and tightness of the lower 
      bound. The authors also present results for the case where the 
      true gradient is not known and the REINFORCE and PGT estimators are used 
      instead. However, it is not clear what these results demonstrate, other 
      than that, unsurprisingly, increasing the number of trajectories used to 
      estimate the gradient improves the gradient estimate and thus learning 
      performance. More worryingly, however, the results also show that even 
      with many trajectories, the errors are quite large, leading to loose 
      bounds that prevent strong performance. This casts doubt on whether the 
      results have practical implications or are only of theoretical interest. 
      Furthermore, the empirical results compare only to fixed-step size 
      variants and not to other baseline methods. In particular, the EM-based 
      policy-gradient-like methods of Kobers & Peters and Vlassis et al. 
      mentioned in the introduction seem like particularly relevant baselines 
      since they do not need to tune a free parameter.