Published as a conference paper at ICLR 2018 DISTRIBUTED DISTRIBUTIONAL DETERMINISTIC POLICY GRADIENTS
This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of N -step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.
A DeepRL algorithm is presented that represents distributions over Q values, as applied to DDPG,
and in conjunction with distributed evaluation across multiple actors, prioritized experience replay, and 
N-step look-aheads. The algorithm is called Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG.
SOTA results are generated for a number of challenging continuous domain learning problems,
as compared to benchmarks that include DDPG and PPO, in terms of wall-clock time, and also (most often) in terms
of sample efficiency.
pros/cons
+ the paper provides a thorough investigation of the distributional approach, as applied to difficult continuous
  action problems, and in conjunction with a set of other improvements (with ablation tests)
- the story is a bit mixed in terms of the benefits, as compared to the non-distributional approach, D3PG
- it is not clear which of the baselines are covered in detail in the cited paper:
  "Anonymous. Distributed prioritized experience replay. In submission, 2017.", 
   i.e., should readers assume that D3PG already exists and is attributable to this other submission?
Overall, I believe that the community will find this to be interesting work.
Is a video of the results available?
It seems that the distributional model often does not make much of a difference, 
as compared to D3PG non-prioritized.  However, sometimes it does make a big difference, i.e., 3D parkour; acrobot.
Do the examples where it yields the largest payoff share a particular characteristic?
The benefit of the distributional models is quite different between the 1-step and 5-step versions. Any ideas why?
Occasionally, D4PG with N=1 fails very badly, e.g., fish, manipulator (bring ball), swimmer.
Why would that be? Shouldn't it do at least as well as D3PG in general?
How many atoms are used for the categorical representation?
As many as [Bellemare et al.], i.e., 51 ?
How much "resolution" is necessary here in order to gain most of the benefits of the distributional representation?
As far as I understand, V_min and V_max are not the global values, but are specific to the current distribution.
Hence the need for the projection. Is that correct?
Would increasing the exploration noise result in a larger benefit for the distributional approach?
Figure 2: DDPG performs suprisingly poorly in most examples. Any comments on this,
or is DDPG best avoided in normal circumstances for continuous problems? :-)
Is the humanoid stand so easy because of large (or unlimited) torque limits?
The wall-clock times are for a cluster with K=32 cores for Figure 1?
"we utilize a network architecture as specified in Figure 1 which processes the terrain info in order to reduce its dimensionality"
Figure 1 provides no information about the reduced dimensionality of the terrain representation, unless I am somehow failing to see this.
"the full critic architecture is completed by attaching a critic head as defined in Section A"
I could find no further documenation in the paper with regard to the "head" or a separate critic for the "head".
It is not clear to me why multiple critics are needed.
Do you have an intuition as to why prioritized replay might be reducing performance in many cases?