State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value. We show that such smoothed Q-values still satisfy a Bellman equation, making them learnable from experience sampled from an environment. Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function. Based on these relationships we develop new algorithms for training a Gaussian policy directly from a learned smoothed Q-value approximator. Our approach is amenable to proximal optimization techniques by augmenting the objective with a penalty on KLdivergence from a previous policy. We find that the ability to learn both a mean and covariance during training allows this approach to achieve much better results on standard continuous control benchmarks.
This paper explores the idea of using policy gradients to learn a stochastic policy on complex control problems.  The central idea is to frame learning in terms of a new kind of Q-value that attempts to smooth out Q-values by framing them in terms of expectations over Gaussian policies.
To be honest, I didn't really "get" this paper.
* As far I understand, all of the original work policy gradients involved stochastic policies.  Many are/were Gaussian.
* All Q-value estimators are designed to marginalize out the randomness in these stochastic policies.
* As far as I can tell, this is equivalent to a slightly different formulation, where the agent emits a deterministic action (\mu,\Sigma) and the environment samples an action from that distribution.  In other words, it seems that if we just draw the box a bit differently, the environment soaks up the nondeterminism, instead of needing to define a new type of Q-value.
Ultimately, I couldn't discern /why/ this was a significant advance for RL, or even a meaningful new perspective on classic ideas.
I thought the little 2-mode MOG was a nice example of the premise of the model.
While I may or may not have understood the core technical contribution, I think the experiments can be critiqued: they didn't really seem to work out.  Figures 2&3 are unconvincing - the differences do not appear to be statistically significant.  Also, I was disappointed to see that the authors only compared to DDPG; they could have at least compared to TRPO, which they mention.  They dismiss it by saying that it takes 10 times as long, but gets a better answer - to which I respond, "Very well, run your algorithm 10x longer and see where you end up!"  I think we need to see a more compelling demonstration of why this is a useful idea before it's ready to be published.
The idea of penalizing a policy based on KL-divergence from a reference policy was explored at length by Bert Kappen's work on KL-MDPs.  Perhaps you should cite that?