Just wanted to share an attempt at implementing in Pytorch the Categorical DQN algorithm published by DeepMind last week in A distributional perspective on reinforcement learning 113.
It currently trains well on Catch, still running experiments on Atari. I am yet to find a good set of hyperparams for Atari games, will post updates about this.
You can find the repo here 277, let me know if you have any suggestions / comments.