https://github.com/ikostrikov/pytorch-a2c-ppo-acktr: 49 an implementation that seems to work properly!
Now I can go back to this project and check what was wrong with my code.
Thanks and Congrats @Ilya_Kostrikov!