Learning Continuous Control Policies by Stochastic Value Gradients
We present a unified framework for learning continuous control policies using backpropagation. It supports stochastic control by treating stochasticity in the Bellman equation as a deterministic function of exogenous noise. The product is a spectrum of general policy gradient algorithms that range from model-free methods with value functions to model-based methods without value functions. We use learned models but only require observations from the environment instead of observations from model-predicted trajectories, minimizing the impact of compounded model errors. We apply these algorithms first to a toy stochastic control problem and then to several physics-based control problems in simulation. One of these variants, SVG(1), shows the effectiveness of learning models, value functions, and policies simultaneously in continuous domains.
The paper introduces a formalizm which enables that allows extending value-based      policy gradient methods to stochastic policies in stochastic     environments. The formalism is based on reparametrization (modeling     stochasticity as a deterministic function of exogenuous noise).     The paper presents algorithms based on this formalism and test them on      multiple control problems.      The problem of extending value-gradient methods to stochastic policies     and environments seems important. The experimental part of the paper      covers different challenging control domains. The main issue with the      paper is that the presentation needs significant improvement to be accessible     to a general RL audience, which is a reasonable requirement from an RL     paper. Specifically, terms are used without being defined first, many     descriptions are informal where formality is required, and many critical     details are missing.This makes it hard to follow and evaluate the exact     contributions of the paper for people outside the community that works on      RL for robotics in continuous state/action spaces.      Details:     - the abstract is currently a bit misleading: it claims to apply     algorithms to "a range of difficult physical control problems", without     mentioning that they are *simulated* control problems. The authors should     clarify this point explicitely in the abstract in a final version of the     paper, if accepted.       line 16 - long sentence, needs break.      line 38 - what do you mean by 'chain' - please be more formal      line 40 - 'backpropagating' has different meanings - please     explain which one of them you refer to here. What is being backpropagated?      line 41 - 'replace the derivatives' - replace them where?      line 51, 55 - what do you mean by 'unroll', please clarify      line 56 - 'learned jointly' - why is it new, many model-based     algorithms can be considered to learn jointly the model,     value-function and policy      line 57 - long sentence, break      line 60-61 - the sentence is vague - please be more explicit      line 76 - Define MDP as a tuple (S, A, P, R, gamma, s0)      line 78 - please clarify why learning time-invariant policies if     reward is time-varying      line 82 - define gamma mathematically (instead of 'strictly less than...')      Equation 1 - missing upper limit in the summation      line 92-94 - definition of V' is confusing, also you might want to use ':="      line 105 - you differentiate with respect to state and theta (not action)      Eq. 3-4 - please provide more details on how you perform the derivation here      line 108 - 'purely model-based formalism' expression is used but was never defined here      line 108 - earlier, did you mean by 'backpropagating' to     backpropagation of the derivatives described here? if yes,     please say explicitely      line 111 - 'unrolling' was never defined      lines 111-114 are unclear      line 117 - 'model' used but never defined      line 118 - how do you define 'modeling errors' and how     sensitive V_theta to them ('highly sensitive' is not informative     enough)      line 119 - 'we start instead' - instead of what?      line 122 - 'real, not planned' - what do you mean by 'planned' - please define      line 124 - 'critics' used but never defined in this context      lines 125-127 - sentence is unclear      line 131 - 'backpropagate efficiently' - what do you mean by saying     that: what is your efficiency measure      line 136 - 'we consider conditional densities' - what do you mean by 'consider'      line 148 - what do you mean by 'exploits' the Jacobian      line 165 - 'standard planning method' - can you provide reference?     please define 'planning' in this context?      Eq 10-11 - you use notation that is not explained: what is '|_\eta,\ksi'      line 185 - 'we can now replace..' - why you can do that?      line 186 - why this case is 'important'      line 187 - \Sigma was not defined      Sec 4.2 - Since it doesn't contain any details, consider moving it to     forward near the place you provide more details      line 218 - 'the parameters of the critic' used but were never defined here      line 257 - 'our framework easily supports' - what do you mean by 'easily'      Sec 6 - it would be good to mention a relevant figure first (e.g. Fig 2) and      then describe in detail what it shows.      Section 6.1 - based on what you determine that [25] is     state-of-the-art? also is it state-of-the-art on a given domain?     multiple domains?