Total stochastic gradient algorithms and applications in reinforcement learning
Backpropagation and the chain rule of derivatives have been prominent; however, the total derivative rule has not enjoyed the same amount of attention. In this work we show how the total derivative rule leads to an intuitive visual framework for creating gradient estimators on graphical models. In particular, previous "policy gradient theorems" are easily derived. We derive new gradient estimators based on density estimation, as well as a likelihood ratio gradient, which "jumps" to an intermediate node, not directly to the objective function. We evaluate our methods on model-based policy gradient algorithms, achieve good performance, and present evidence towards demystifying the success of the popular PILCO algorithm [5].
In this paper, the authors investigate the application of a total gradient theorem to the evaluation of gradient of expectations in probabilistic computation graphs. By combining the total gradient rule with other known gradient estimators (reparametrization and score function estimator), they derive new estimators, in particular in the context of policy gradient for reinforcement learning. They use these new estimators to explain the empirical success of PILCO.  Though I found the exposition a bit lacking, I found the paper overall very interesting (the Gaussian shaping gradient in particular); both because the problem of estimating gradients of expectations is fundamental (and therefore new estimators are highly valuable) and for the careful analysis in the experimental section.  - I felt that what the authors really wanted to present was derivative of marginal distribution with respect to other marginal distributions, but because this would have been mathematically cumbersome, introduced 'virtual' parameters \xi, even though the distribution for which \xi are parameters are never actually specified, and it in the vast majority of PCGs, there would be no known distribution with parameters \xi corresponding to the true marginal distribution. This leads to a total stochastic gradient theorem that takes a strange form, since it involves variables which are both underspecified and never actually computed.  I understand why this was done, but this makes this entire section read a bit strangely. - Theorem 1 - even if it follows from the total derivative rule, deserves a short proof (and the total derivative rule, a reference). - The second and third section don't tie very well together, since one section (section 3) relates gradient of distribution parameters to others, and the other (section 2), is instead about computing gradients of expectations. Of course, the parameters of a distribution can often be written as expectation of sufficient statistics (a fact used by the Gaussian Shaping gradient), but this is never made clear and the sections are not connected enough. For that matter, it's not clear how does the policy gradient (eq 5) actually relates to a total gradient theorem (what is the critic a gradient of?). The connection between (5) and (6) being different estimators of the same computation graph was already known (see for instance [2]).  The connection between path-relative partial gradient and reparametrization is also only implicit in the text.  - Section 5.2 was confusingly written. As I understand, the first half total gradient theorem is applied with variables x_m, and the first part of the gradient is estimated by making a gaussian assumption and noting that the parameters of the marginal gaussian can be written as expectation of sufficient statistics; at which point the score function estimator is applied to compute the expectation gradient.   - For the PCG framework, aren't properties 1 and 2 implying properties 3 and 4?  Minor:  - I didn't feel the introduction of a new framework was necessarily warranted- as stochastic computation graphs were also introduced as factor graphs and the problem of interest (gradient of expectation) was the same. The SCG framework did not explicitly call for sampling the graph forward, though it is true that the methods introduced call for sampling - but practically, so do the methods in this paper. Both frameworks define distributions over variables, identify the relevant problem as an intractable deterministic distribution, and neither paper actually offers fully deterministic computation on those distributions as practical algorithms. The main difference is the explicit focus on computing marginal distributions in the PCG framework. - Given you give variables meaningful names in the PCG for model-free RL, it would be simpler overall not to index the parameters variables \xi, but perhaps to use a notation like \xi(x), \xi(u), etc. \xi(x_m) would be much easier to parse than \xi_m, since that notation could refer to the sufficient statistics of any m-indexed variable. - Hunch: the gaussian shaping gradient could be related to policy gradients for average-cost MDPs [1], since in both cases , the gradient of the expected value is written as gradient of marginal distributions times rewards (instead of gradient of conditional distribution times returns in the classical PG theorem). The settings are different and no Gaussian approximation is made, but this felt related.  [1] Simulation-Based Optimization of Markov Reward Processes, Marbach and Tsitsiklis.  [2] Stochastic Value Gradients, Heess et. Al