We introduce a new approach to estimate continuous actions using actor-critic algorithms for reinforcement learning problems. Policy gradient methods usually predict one continuous action estimate or parameters of a presumed distribution (most commonly Gaussian) for any given state which might not be optimal as it may not capture the complete description of the target distribution. Our approach instead predicts M actions with the policy network (actor) and then uniformly sample one action during training as well as testing at each state. This allows the agent to learn a simple stochastic policy that has an easy to compute expected return. In all experiments, this facilitates better exploration of the state space during training and converges to a better policy.
This paper describes an approach to stochastic control using RL that extends DDPG with a stochastic policy.  A standard DDPG setup is extended such that the actor now produces M actions at each timestep.  Only one of the M actions will be executed in the environment using a uniform sampling.  The sampled action is the only that will receive a gradient update from the critic network. The authors demonstrate that such a stochastic policy performs better on average in a series of benchmark control tasks.
I find the general idea of the work compelling, but the particular approach is rather poor.  The fact that we are choosing the number of modes in the uniform distribution is a bit underwhelming (a more compelling architecture could have proposed a policy conditioned on gaussian noise for example, thus having better coverage of the distribution). I found the proposed apprach to be under-analyzed and the stochastic aspects of the policy are undervalued.   The main claim being argued in the paper is that the proposed stochastic policy has better final performance on average than a deterministic policy, but the only practical difference seems to be a slightly more structured approach to exploration.  
However, very little attention is paid to trying different exploration methods with the deterministic policy (incidentally, Ornstein-Uhlenbeck process noise is not something I'm familiar with, a citation to the use of this noise for exploration as well as a more explicit explanation would have been appreciated).  One interpretation is that each of the M sub-policies follows a different mode of the Q-value distribution over the action space.  But is this indeed the case?  There is a brief analysis of this with cartpole, but a more complete look at how actions are clustered in the action space would make this paper much more compelling.  Even in higher-dimensional action spaces, you could look at a t-SNE projection or cluster analysis to try and see how many modes the agent is reasoning over.  Additionally, the baseline agent should have used additional exploration methods as these can quickly change the performance of the agent.
I also think that better learning is not the only redeeming aspect of a stochastic policy.  In the face of a non-stationary environment, a stochastic policy will likely be much more robust.  Additionally, it will have much better performance against adversarial environments.  Given the remaining available space in the paper it would have been interesting to provide more insight into the proposed methods gains in these areas.